Kim Berg Hansen on “Use Cases of Row Pattern Matching in Oracle 12c”

As I write this, I am listening to Kim Berg Hansen explain the MATCH_RECOGNIZE clause. He was kind enough to give me credit for some of the examples and mention this blog.

In addition to my blog posts on the subject, you may enjoy my presentation on SlideShare. Please download it to see the animations!

If you have questions arising from the presentation, please add a comment here.

Bravo to Kim for his remarkably interactive webinar!

Chunking tables 8: second solution

Here is the second complete solution to the problem of dividing a table into chunks of equal size. This is the solution whose first version was marked “correct” on OTN. It is the basis for what Bryn Llewellyn calls “Approx_Method_Sql_Ashton_2” in his white paper.

Current list of posts: 

Overview

  1. extents_data: get all extents and data object ids for the table
  2. extents_with_sums:
    1. treat all the table blocks as if they were one contiguous set
    2. number all the blocks in ascending sequence
    3. calculate total number of blocks
  3. filtered_extents: filter out the extents that do not contain a chunk boundary
  4. expanded_extents: expand every row that crosses chunk boundaries so that there will be one row per chunk
  5. rowids:
    1. calculate each ROWID
    2. group by chunk.

filtered_extents

with extents_data as (
  select /*+ qb_name(extents_data) */
    o.data_object_id, e.file_id, e.block_id, e.blocks
  from dba_extents e
  join all_objects o
  on e.owner = o.owner
    and e.segment_name = o.object_name
    and e.segment_type = o.object_type
    and decode(e.partition_name, o.subobject_name, 0, 1) = 0
  where e.segment_type like 'TABLE%'
    and e.owner = :owner
    and e.segment_name = :table_name
)
, extents_with_sums as (   
  select /*+ qb_name(extents_with_sums) */
    sum(blocks) over() as total_blocks, 
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) - blocks as first_extent_block,   
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) as next_extent_block, 
    e.* 
  from extents_data e 
) 
, filtered_extents as (  
  select /*+ qb_name(filtered_extents)*/ * from (
    select
      width_bucket(first_extent_block-1, 0, total_blocks, :num_chunks)
        as prev_chunk, 
      width_bucket(first_extent_block, 0, total_blocks, :num_chunks)
        as first_chunk, 
      width_bucket(next_extent_block-1, 0, total_blocks, :num_chunks)
        as last_chunk,  
      width_bucket(next_extent_block, 0, total_blocks, :num_chunks)
        as next_chunk, 
      e.* 
    from extents_with_sums e   
  )
  where prev_chunk < next_chunk
)

The first two named subqueries are identical to the first solution. The filtered_extents subquery differs only by calculating more columns, since I will need them all later:

  • PREV_CHUNK is the chunk to which the end of the previous extent belongs.
  • FIRST_CHUNK and LAST_CHUNK are the chunks containing the first and last block of the current extent.
  • NEXT_CHUNK is the chunk to which the start of the next extent belongs.

expanded_extents

, expanded_extents as (
  select /*+ qb_name(expanded_extents) */
    first_chunk + level - 1 as chunk,
    prev_chunk, next_chunk, data_object_id, file_id,
    block_id, total_blocks, first_extent_block
  from filtered_extents
  connect by first_extent_block = prior first_extent_block
    and prior sys_guid() is not null
    and first_chunk + level - 1 <= last_chunk
)

This subquery used the method explained in my previous post to expand each row that crosses chunk boundaries. The only difference among the expanded rows is the chunk number assigned to each.

rowids

, rowids as (
  select /*+ qb_name(rowids) */ chunk,
    case when chunk > prev_chunk then dbms_rowid.rowid_create(
      1, data_object_id, file_id,
      block_id + CEIL( (chunk-1) * total_blocks / :num_chunks - 1 ) + 1 - first_extent_block,
      0
    ) 
    end as start_rowid,
    case when chunk < next_chunk then dbms_rowid.rowid_create(
      1, data_object_id, file_id,
      block_id + CEIL( chunk * total_blocks / :num_chunks - 1 )  - first_extent_block,
      32767
    ) 
    end as end_rowid
  from expanded_extents
)
select chunk, min(start_rowid) start_rowid, max(end_rowid) end_rowid
from rowids
group by chunk;

This subquery calculates the starting and ending rowids for each chunk. It is much more compact than the way I did this previously.

By comparing the current chunk number with prev_chunk and next_chunk, I can tell if I should calculate a starting rowid, an ending rowid or both. To calculate the block number,

  • I start from the block_id of the extent
  • I add the block number of the target chunk boundary
  • then I subtract the number I assigned previously to the first block of the extent.

At this point, each chunk may be present in one row or two. A simple GROUP BY gives me one row per chunk.

Evolution

My version of this solution has changed over time, especially while writing this series.

  • I now make sure my numbering scheme always starts with zero.
  • I have also incorporated WIDTH_BUCKET() and changed the way I calculate chunk boundaries to be coherent.
  • Above all, I have simplified the code that says when to calculate the rowids, so that the KEEP (DENSE_RANK...) functions are no longer necessary. I apologize for the earlier complicated code.

By the way, here is the code all together if anyone wants to copy and paste.

with extents_data as (
  select /*+ qb_name(extents_data) */
    o.data_object_id, e.file_id, e.block_id, e.blocks
  from dba_extents e
  join all_objects o
  on e.owner = o.owner
    and e.segment_name = o.object_name
    and e.segment_type = o.object_type
    and decode(e.partition_name, o.subobject_name, 0, 1) = 0
  where e.segment_type like 'TABLE%'
    and e.owner = :owner
    and e.segment_name = :table_name
)
, extents_with_sums as (   
  select /*+ qb_name(extents_with_sums) */
    sum(blocks) over() as total_blocks, 
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) - blocks as first_extent_block,   
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) as next_extent_block, 
    e.* 
  from extents_data e 
) 
, filtered_extents as (  
  select /*+ qb_name(filtered_extents)*/ * from (
    select
      width_bucket(first_extent_block-1, 0, total_blocks, :num_chunks)
        as prev_chunk, 
      width_bucket(first_extent_block, 0, total_blocks, :num_chunks)
        as first_chunk, 
      width_bucket(next_extent_block-1, 0, total_blocks, :num_chunks)
        as last_chunk,  
      width_bucket(next_extent_block, 0, total_blocks, :num_chunks)
        as next_chunk, 
      e.* 
    from extents_with_sums e   
  )
  where prev_chunk < next_chunk
)
, expanded_extents as (
  select /*+ qb_name(expanded_extents) */
    first_chunk + level - 1 as chunk,
    prev_chunk, next_chunk, data_object_id, file_id,
    block_id, total_blocks, first_extent_block
  from filtered_extents
  connect by first_extent_block = prior first_extent_block
    and prior sys_guid() is not null
    and first_chunk + level - 1 <= last_chunk
)
, rowids as (
  select /*+ qb_name(rowids) */ chunk,
    case when chunk > prev_chunk then dbms_rowid.rowid_create(
      1, data_object_id, file_id,
      block_id + CEIL( (chunk-1) * total_blocks / :num_chunks - 1 ) + 1 - first_extent_block,
      0
    ) 
    end as start_rowid,
    case when chunk < next_chunk then dbms_rowid.rowid_create(
      1, data_object_id, file_id,
      block_id + CEIL( chunk * total_blocks / :num_chunks - 1 )  - first_extent_block,
      32767
    ) 
    end as end_rowid
  from expanded_extents
)
select chunk, min(start_rowid) start_rowid, max(end_rowid) end_rowid
from rowids
group by chunk;

Chunking tables 7: prior sys_guid() ???

The second solution for dividing a table into equal chunks does not do a JOIN. Instead it expands extent rows that contain several chunk boundaries, using an obscure method that I need to explain.

Current list of posts: 

Generating rows

To generate rows quickly, it is common to use the CONNECT BY clause.

select level num
from dual
connect by level <= 5;
NUM
1
2
3
4
5

 

This is a simple “hierarchical” query that uses the LEVEL pseudocolumn. LEVEL starts at 1 and increments every time the CONNECT BY is executed. The CONNECT BY condition says to keep going as long as LEVEL is less than or equal to 5.

Note that there is no reference to a PRIOR row in the condition, since there is only one row in the DUAL table.

Expanding rows

Here is where things get tricky.

Suppose I have two rows and I want to expand both of them to get one row for every integer in the range. For example, take this table:

drop table U purge;

create table U as
select 1 range_id, 2 range_end from dual
union all
select 2, 3 from dual;

select * from u;
RANGE_ID RANGE_END
1 2
2 3

 

Using the very same technique as before:

select range_id, range_end, level
from u
connect by level <= range_end;
RANGE_ID RANGE_END LEVEL
1 2 1
1 2 2
2 3 3
2 3 2
2 3 3
2 3 1
1 2 2
2 3 3
2 3 2
2 3 3

 

What is this mess? It looks like I’m starting with each row and connecting to the other row – which makes sense because I’m not saying to stay on the same row. Let’s try again:

select range_id, range_end, level
from u
connect by level <= range_end
and range_id = prior range_id

Error report - SQL Error: ORA-01436: CONNECT BY loop in user data

Now I made a reference to something PRIOR – the range_id. Oracle sees that the same range_id is accessed twice in a row, so it assumes there is an infinite loop and aborts the execution.

There is a way to avoid that error, using the NOCYCLE keyword:

select range_id, range_end, level
from u
connect by nocycle level <= range_end
and range_id = prior range_id;
RANGE_ID RANGE_END LEVEL
1 2 1
2 3 1

 

Well, I didn’t get the error, but Oracle still considers that doing the same range_id twice would be a loop, so it stops first.

Enter SYS_GUID()

What we need is to add something to the prior row that will make Oracle think it is different. SYS_GUID() is a very low-cost function that returns a nonrepeating value. If we refer to PRIOR SYS-GUID() in a condition, that is enough to make the prior row unique and to prevent the perception of an infinite loop.

select range_id, range_end, level
from u
connect by level <= range_end
and range_id = prior range_id
and prior sys_guid() is not null;
RANGE_ID RANGE_END LEVEL
1 2 1
1 2 2
2 3 1
2 3 2
2 3 3

 

In the next post, I will use this technique to expand extent rows that contain multiple chunks.

P.S. If you have version 12c, you can use the LATERAL clause, which will expand each row without bending your mind.

select * from u,
lateral(
  select level lvl from dual
  connect by level <= range_end
);

Chunking tables 6: JOIN solution

At last, here is the first complete solution to the problem of dividing a table into 12 chunks with equal numbers of blocks. James Su and I proposed this solution on OTN, but without the “filtered_extents” optimisation.

The OTN solution is the basis for what Bryn Llewellyn calls “Approx_Method_Sql_Ashton_1” in his white paper.

Current list of posts: 

Preparation

drop table t purge;

create table t as
select rpad('x', 4000, 'x') x
from dual
connect by level <= 130;

var owner varchar2(128);
var table_name varchar2(128);
var num_chunks number;
begin
  :owner := user;
  :table_name := 'T';
  :num_chunks := 12;
end;
/

Up to now I have been hard-coding the owner name, the table name and the number of chunks (12). I hoped this would make the explanation a bit more concrete and therefore easier to follow. Now it’s time to remember that such values should be held in variables, both as a good programming practice and to allow Oracle to reuse the SQL statement without having to parse it every time. Here I am using “host variables”, because it is easier to test the query bit by bit when it is not wrapped in PL/SQL. Ultimately this SQL should live inside a PL/SQL function with formal parameters.

Overview

  1. extents_data: get all extents and data object ids for the table
  2. extents_with_sums:
    1. treat all the table blocks as if they were one contiguous set;
    2. number all the blocks in ascending sequence
    3. calculate total number of blocks
  3. filtered_extents: filter out the extents that do not contain a chunk boundary
  4. chunk_boundaries: calculate the chunk boundaries
  5. rowids:
    1. Use JOIN to map chunk boundaries to extents;
    2. calculate each ROWID
    3. group by chunk

extents_data

with extents_data as (
  select /*+ qb_name(extents_data) */
    o.data_object_id, e.file_id, e.block_id, e.blocks
  from dba_extents e
  join all_objects o
  on e.owner = o.owner
    and e.segment_name = o.object_name
    and e.segment_type = o.object_type
    and decode(e.partition_name, o.subobject_name, 0, 1) = 0
  where e.segment_type like 'TABLE%'
    and e.owner = :owner
    and e.segment_name = :table_name
)

I wrote a series of named subqueries using the WITH() clause, and gave a “query block name” (qb_name) equal to the subquery name in order to better understand the execution plan. It turns out that the optimizer rewrites the query in such a way that some of the qb_names disappear.

This first clause provides the raw material for calculating the ROWIDs we need: the data object ids, the file ids, the block ids and the number of blocks per chunk.

extents_with_sums

, extents_with_sums as (
  select /*+ qb_name(extents_with_sums) */
    sum(blocks) over() as total_blocks,
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) - blocks as first_extent_block,
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) - 1 as last_extent_block,
    e.*
  from extents_data e
) 

This subquery gets the total number of blocks (available in each row of the output), plus two running totals:

  • The number of blocks up to but not including the current extent.
    Since I am numbering the blocks consecutively starting from zero, this is the number assigned to the first block of the current extent.
  • The number of blocks up to and including the current extent, minus one.
    This is the number assigned to the last block of the current extent.

filtered_extents

, filtered_extents as (
  select /*+ qb_name(filtered_extents) */ * from (
    select
      width_bucket(first_extent_block-1, 0, total_blocks, :num_chunks)
        as prev_chunk,
      width_bucket(last_extent_block+1, 0, total_blocks, :num_chunks)
        as next_chunk,
      e.*
    from extents_with_sums e
  )
  where prev_chunk < next_chunk
)

Here I eliminate extents that do not contain any chunk boundaries.

  • WIDTH_BUCKET() tells me to which chunk a block belongs.
  • PREV_CHUNK is the chunk to which the end of the previous extent belongs.
  • NEXT_CHUNK is the chunk to which the start of the next extent belongs.
  • If they are the same, then the current extent is in the middle of a chunk and I can safely eliminate it.

For details on how I use WIDTH_BUCKET() here, and CEIL() in the next section, click here: https://stewashton.wordpress.com/2016/02/04/chunking-tables-4-histograms/

chunk_boundaries

, chunk_boundaries as (
  select  /*+qb_name(chunk_boundaries) */
    level chunk,
    CEIL( (level-1) * total_blocks / :num_chunks - 1 ) + 1 first_chunk_block,
    CEIL(  level    * total_blocks / :num_chunks - 1 )     last_chunk_block
  from (select total_blocks from extents_with_sums where rownum = 1)
  connect by level <= least(:num_chunks, total_blocks)
)

Having the extents and the extent boundaries, I now need the chunk boundaries.

  • I get TOTAL_BLOCKS by querying the first row from the EXTENTS_WITH_SUMS subquery.
  • Each chunk has a range equal to TOTAL_BLOCKS / :NUM_CHUNKS
  • The upper boundary of each chunk is LEVEL times that range
  • I need the last block that is less than the upper boundary, so I subtract 1 and then use CEIL().
  • least(:num_chunks, total_blocks) ensures that the number of chunks cannot be more than the total number of blocks. This is an alternative to raising an exception.

rowids

, rowids as (
  select  /*+qb_name(rowids) */ chunk,
    case when b.first_chunk_block
      between e.first_extent_block and e.last_extent_block then
        dbms_rowid.rowid_create(
          1, data_object_id, file_id,
          block_id + first_chunk_block - first_extent_block, 0
        )
    end as start_rowid,
    case when b.last_chunk_block
      between e.first_extent_block and e.last_extent_block then
        dbms_rowid.rowid_create(
          1, data_object_id, file_id,
          block_id + last_chunk_block - first_extent_block, 32767
        )
    end as end_rowid
  from chunk_boundaries b
  join filtered_extents e
  on b.first_chunk_block between e.first_extent_block and e.last_extent_block
   or b.last_chunk_block between e.first_extent_block and e.last_extent_block
)
select chunk, max(start_rowid) start_rowid, max(end_rowid) end_rowid
from rowids
group by chunk;

This is the heart of the solution. By joining the extents to the chunk boundaries, I automatically expand any extent that spans more than one chunk. If I ask for 12 chunks, the JOIN will return between 12 and 24 rows. I calculate START_ROWID only when appropriate, otherwise it is null – and the same goes for END_ROWID.

Finally, I condense the result into 12 rows using GROUP BY.

Evolution

My version of this solution has changed over time.

  • On OTN and in Bryn Llewellyn’s white paper, I did not filter the extents.
  • I now make sure my numbering scheme always starts with zero.
  • I have also incorporated WIDTH_BUCKET() and changed the way I calculate chunk boundaries to be coherent.

Next, another solution, this time without a join.

By the way, here is the code all together if anyone wants to copy and paste.

with extents_data as (
  select /*+ qb_name(extents_data) */
    o.data_object_id, e.file_id, e.block_id, e.blocks
  from dba_extents e
  join all_objects o
  on e.owner = o.owner
    and e.segment_name = o.object_name
    and e.segment_type = o.object_type
    and decode(e.partition_name, o.subobject_name, 0, 1) = 0
  where e.segment_type like 'TABLE%'
    and e.owner = :owner
    and e.segment_name = :table_name
)
, extents_with_sums as (
  select /*+ qb_name(extents_with_sums) */
    sum(blocks) over() as total_blocks,
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) - blocks as first_extent_block,
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) - 1 as last_extent_block,
    e.*
  from extents_data e
)
, filtered_extents as (
  select /*+ qb_name(filtered_extents) */ * from (
    select
      width_bucket(first_extent_block-1, 0, total_blocks, :num_chunks)
        as prev_chunk,
      width_bucket(last_extent_block+1, 0, total_blocks, :num_chunks)
        as next_chunk,
      e.*
    from extents_with_sums e
  )
  where prev_chunk < next_chunk
)
, chunk_boundaries as (
  select  /*+qb_name(chunk_boundaries) */
    level chunk,
    CEIL( (level-1) * total_blocks / :num_chunks - 1 ) + 1 first_chunk_block,
    CEIL(  level    * total_blocks / :num_chunks - 1 )     last_chunk_block
  from (select total_blocks from extents_with_sums where rownum = 1)
  connect by level <= least(:num_chunks, total_blocks)
)
, rowids as (
  select  /*+qb_name(rowids) */ chunk,
    case when b.first_chunk_block
      between e.first_extent_block and e.last_extent_block then
        dbms_rowid.rowid_create(
          1, data_object_id, file_id,
          block_id + first_chunk_block - first_extent_block, 0
        )
    end as start_rowid,
    case when b.last_chunk_block
      between e.first_extent_block and e.last_extent_block then
        dbms_rowid.rowid_create(
          1, data_object_id, file_id,
          block_id + last_chunk_block - first_extent_block, 32767
        )
    end as end_rowid
  from chunk_boundaries b
  join filtered_extents e
  on b.first_chunk_block between e.first_extent_block and e.last_extent_block
   or b.last_chunk_block between e.first_extent_block and e.last_extent_block
)
select chunk, max(start_rowid) start_rowid, max(end_rowid) end_rowid
from rowids
group by chunk;

Chunking tables 5: chunk boundaries

We still want to split a table into 12 chunks with equal numbers of blocks, so we can access each chunk using a rowid range. We have the extents we need and now we need to calculate the exact chunk boundaries.

Current list of posts: 

Histograms and WIDTH_BUCKET

I explained in my previous post that WIDTH_BUCKET is based on a histogram having identical intervals in a range.

  • The minimum range value is included in the range but the maximum range value is not.
  • The same thing is true for each interval: the minimum value is included in the interval, but not the maximum value.

We have already used WIDTH_BUCKET to answer the question: “what chunk is this block in?” Now we are asking the opposite question: “what blocks start and end this chunk?” We need to make absolutely sure the answers are coherent!

Getting the upper chunk boundary

Using our test table from the previous post, let’s compare what WIDTH_BUCKET tells us with a wrong answer and a right answer. Then I’ll explain in detail. Right now, I’m only interested in those blocks that end a chunk.

select this_block_chunk, block last_block,
  round(256 * (this_block_chunk / 12), 1) max_chunk_val,
  floor(256 * (this_block_chunk / 12)) floor_val,
  ceil (256 * (this_block_chunk / 12) - 1) right_val
from (
  select level-1 block,
  width_bucket(level-1,0,256,12) this_block_chunk,
  width_bucket(level,0,256,12) next_block_chunk
  from dual
  connect by level <= 256
)
where this_block_chunk < next_block_chunk;
THIS_BLOCK_CHUNK LAST_BLOCK MAX_CHUNK_VAL FLOOR_VAL RIGHT_VAL
1 21 21.3 21 21
2 42 42.7 42 42
3 63 64 64 63
4 85 85.3 85 85
5 106 106.7 106 106
6 127 128 128 127
7 149 149.3 149 149
8 170 170.7 170 170
9 191 192 192 191
10 213 213.3 213 213
11 234 234.7 234 234
12 255 256 256 255

 

For each block, I asked WIDTH_BUCKET what chunk it is in and what chunk the next block is in, and I kept only those blocks where the next block is in the next chunk. That means I have a list of the last blocks for each chunk.

In the third column, I calculate the maximum value for each chunk. Notice it is not always an integer value. How can we calculate the greatest block number that is less than that maximum value?

  • Using CEIL() or ROUND(), we will sometimes get a result that is greater than the maximum value.
  • Using FLOOR(), we will sometimes get a result that is equal to the maximum value – but we need to be less than, not equal to! You can see the problem in chunks 3, 6, 9 and 12.
  • The right answer is to subtract 1 from the maximum value and then use CEIL(). That way the result will always be less than the maximum value, but as close to it as possible.

Getting the lower chunk boundary

I’ll bet you thought of this yourself. The lower boundary of a chunk is one greater that the upper boundary of the previous chunk :-)

Coming next…

We have the extents we need, and we have a method for calculating all the chunk boundaries. In the next post, I’ll present the first of two complete solutions for chunking a table.

Current list of posts: 

Chunking tables 4: Histograms

After “only” 3 posts on chunking tables, we now have the raw materials for breaking up a table into 12 chunks with equal numbers of blocks, and for getting ROWID ranges that let us access the data. Now let’s chunk!

Current list of posts:

What needs to be done?

The basic idea is pretty simple:

  1. Treat all the table blocks as if they were one contiguous set
  2. Number all the blocks in ascending sequence
  3. Filter out the extents that do not contain a chunk boundary
    (yes, I do this before the next step!)
  4. Calculate the chunk boundaries
  5. Map each chunk boundary to the corresponding block in the right extent
  6. Calculate each ROWID

One contiguous set

Blocks within an extent are contiguous, at least as seen by us. Blocks in different extents may not be contiguous; they may even live in different files or data objects. We don’t care about any of that as long as our ROWID ranges work. For that, we just need to order the “raw material” by data_object_id, file_id and block_id. That way all the extents (and the blocks in the extents) will be ordered by ascending ROWID value.

Number the blocks in ascending sequence

For this we can use the analytic SUM() function to get the running sum of blocks as we go through the extents in the correct order.

One more thing: to calculate the chunk boundaries, we are going to need the total number of blocks in the table. A different analytic SUM() function will provide that total.

To better demonstrate, I’m going to create a bigger table.

create table t as
select rpad('x', 4000, 'x') x
from dual
connect by level <= 130;

with extents_data as (
  select /*+ qb_name(extents_data) */
    o.data_object_id, e.file_id, e.block_id, e.blocks
  from dba_extents e
  join all_objects o
  on e.owner = o.owner
    and e.segment_name = o.object_name
    and e.segment_type = o.object_type
    and decode(e.partition_name, o.subobject_name, 0, 1) = 0
  where e.segment_type like 'TABLE%'
    and e.owner = user
    and e.segment_name = 'T'
)
, extents_with_sums as (   
  select /*+ qb_name(extents_with_sums) */
    sum(blocks) over() as total_blocks, 
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) - blocks as first_extent_block,   
    sum(blocks) over(
      order by data_object_id, file_id, block_id
    ) as next_extent_block, 
    e.*
  from extents_data e 
) 
select * from extents_with_sums;
TOTAL_BLOCKS FIRST_EXTENT_BLOCK NEXT_EXTENT_BLOCK BLOCK_ID BLOCKS
256 0 8 128 8
256 8 16 136 8
256 16 24 144 8
256 24 32 152 8
256 32 40 160 8
256 40 48 168 8
256 48 56 176 8
256 56 64 184 8
256 64 72 192 8
256 72 80 200 8
256 80 88 208 8
256 88 96 216 8
256 96 104 224 8
256 104 112 232 8
256 112 120 240 8
256 120 128 248 8
256 128 256 256 128

I removed two columns from the output to make it fit better.

There are a lot of things to notice here, so bear with me:

  • I break the entire SQL query into a series of subqueries using WITH clauses.
  • Each WITH clause gets a “query block name” (qb_name) equal to the subquery name. I hope this will make it easier to analyze the execution plan.
  • In the numbering scheme, I chose to start with 0 and not with 1 because the math is a bit easier.
  • SUM(blocks) OVER() gives me the total number of blocks on every line.
  • Since I start with block 0, The running SUM() of blocks actually gives me the block number of the beginning of the next extent. To get the beginning of this extent, I just subtract the current number of blocks.
    (I could use a fancy “windowing clause” in the analytic function, but this seemed easier to explain.)
  • Since I am numbering every block consecutively, I can calculate the last block of any extent: it is one less than the first block of the following extent.
  • Since I am using ASSM, the last extent jumped from 8 blocks to 128 blocks.

Did I say histograms?

A histogram is a range divided into intervals that have identical size. Isn’t that exactly what we want? The range goes from 0 (the first block) up to but not including TOTAL_BLOCKS, and the number of intervals is the number of chunks, which is 12 in our example. To make sure no block gets put in two chunks, each interval includes the lower boundary and excludes the upper boundary. This is called a “closed-open” interval.

There is a function that does exactly what I just described:
WIDTH_BUCKET(expression, included_min_value, excluded_max_value, number_of_intervals)
or in this case
WIDTH_BUCKET(block, 0, 256, 12).

Let’s take every block from 0 to 255 and find out what chunk it is in.

select chunk, min(n) first_block, max(n) last_block,
max(n) + 1 - min(n) num_blocks
from (
  select level-2 n, 
  width_bucket(level-2,0,256,12) chunk
  from dual
  connect by level <= 258
)
group by chunk
order by chunk;
CHUNK FIRST_BLOCK LAST_BLOCK NUM_BLOCKS
0 -1 -1 1
1 0 21 22
2 22 42 21
3 43 63 21
4 64 85 22
5 86 106 21
6 107 127 21
7 128 149 22
8 150 170 21
9 171 191 21
10 192 213 22
11 214 234 21
12 235 255 21
13 256 256 1

 

I deliberately went outside the boundaries to show that they get put into “illegal” chunks: 0 and 13.

Notice that the number of blocks in each chunk is as equal as possible. This is looking pretty good.

There is one problem though: WIDTH_BUCKET answers the question “what chunk am I in?” but it doesn’t answer the question “what are the lower and upper boundaries of each chunk?” We’ll see how to do that later.

Filter unnecessary extents

This is a little performance boost that Jonathan Lewis praised. A very large table, or a table with lots of partitions, will have many rows in DBA_EXTENTS, but we are only interested in the rows that contain a chunk boundary. In our example, that means 24 rows at the most.

When does an extent not contain a chunk boundary? When the end of the previous extent and the beginning of the next extent both belong to the same chunk. Thanks to WIDTH_BUCKET and the running totals I just calculated, I can filter out those extents right now.

with extents_data as (...)
, extents_with_sums as (...) 
, filtered_extents as (  
  select /*+ qb_name(filtered_extents)*/ * from (
    select
      width_bucket(first_extent_block-1, 0, total_blocks, 12)
        as prev_chunk, 
      width_bucket(next_extent_block, 0, total_blocks, 12)
        as next_chunk, 
      e.* 
    from extents_with_sums e   
  )
  where prev_chunk < next_chunk
)
select prev_chunk, next_chunk,
  first_extent_block, next_extent_block, blocks
from filtered_extents;
PREV_CHUNK NEXT_CHUNK FIRST_EXTENT_BLOCK NEXT_EXTENT_BLOCK BLOCKS
0 1 0 8 8
1 2 16 24 8
2 3 40 48 8
3 4 56 64 8
3 4 64 72 8
4 5 80 88 8
5 6 104 112 8
6 7 120 128 8
6 13 128 256 128

 

If you compare these results with the FIRST_BLOCK and LAST_BLOCK numbers we calculated previously, you can see that the chunk boundaries are all contained within the extents we retained.

Are we done yet?

I wish I could say yes, but I can’t. We have gotten pretty far though. We have the total number of blocks, we have numbered all the blocks consecutively in the right order and we have retained only those extents that contain chunk boundaries.

In the next post I’m going to show how to calculate the chunk boundaries. Afterwards, I’ll demonstrate two methods for mapping those boundaries to extents in order to get the ROWIDs we want.

Current list of posts: