Best practice for Perform Tune values in XPP Job Ticket such as Max Page Size

I want to share some best practice advice for setting the Perform Tune values in the Job Ticket (JT), specifically with the Max Page SizeMax Pages, and Max DLD Entries fields.

I've noticed a trend in customer samples that I've seen where these field values are just "maxed out", i.e. set to their maximum values as sort of a "default" setup of the JT.

For example, I'm seeing a lot of JT's with the Max Page Size field set to 99999999 (the maximum value you can enter in that field) - even when nothing close to that value is needed for the DIV's that belong to those JOB's.

And with the same customer samples I also typically see the Max Pages and Max DLD Entries fields both set to "jumbo" (which is 1000 pages and 1500 entries), or even higher - because some users think that these are the maximum values for these fields. Note: Actually that's not the case, it's only that "jumbo" is the highest pre-defined value for these fields (the max value for these fields is actually 32767 and 262144 - but I'm not sure I should be revealing that here Slight smile).

And just using these "highest" values, by default, for these fields is not a good thing to do.

Why?

Well put simply, it will have an adverse effect on XPP performance and for no really good reason (and could adversely affect the overall performance of the system running XPP as well if other things are being done besides XPP) - hence the naming of these fields as "Perform Tune" (aka performance related).

So, what is the best practice for these fields?

Set them only as high as you need, or foresee that you will need, for the DIV's that will belong to a particular JOB.

For example (with probably the easiest of the fields to determine), if you know that no individual DIV in a JOB is going to have more than 100 pages in it then don't set the Max Pages field (needlessly) to 1000 pages (jumbo). And it may be more challenging to determine the appropriate value to use for the Max Page Size and Max DLD Entries fields, but with experience using XPP you will become pretty proficient at figuring out what values you need.

And there is rarely any lasting ill effects if you do run into the situation of having values that are too small. Yes, you'll get a "fatal" error message such as "Page Buffer Overflow …", "Page ... too large for Page Buffer", or "Number of Pages exceeds maximum ...". But then all you usually need to do is increase the appropriate field in the JT and go back and compose again.

I'm going to stop there for this initial post; many will not want to bother with the details and explanation, but for those that are interested I am going to follow with further "posts" (in this thread) to further explain and even give examples showing the impact.

Jonathan Dagresta
SDL XPP Engineering

Parents
  • Here are more details and an example illustrating the impact of all of this.

    In general there are two "philosophies" in memory allocation for a program's use when dealing with data - "static" versus "dynamic". A "static" approach means that memory is allocated up front (and does not move) and a "dynamic" approach means that memory is allocated (and reallocated) as needed. But typically when memory is reallocated (to a bigger size), that also means that the location of that memory (within a program's address space) has moved to a new location.

    For some of the larger chunks of data associated with a DIV, from the beginning XPP has used a "static" type of memory allocation. These include the memory for "Page Buffers" (used to hold the contents of pages - Max Page Size), the memory used to hold the list of pages (corresponding to the job.dpd file - Max Pages), and the memory used to hold the list of "named" items such as footnotes and pickups (corresponding to the job.dld file - Max DLD Entries).

    There are a number of reasons for having done this, and although some of those reasons might not be as important as they were at the "beginning" of XPP there are still some good reasons for keeping this approach. A big one is still the speed with which XPP is able to do very complicated composition calculations including things like laying out pages, horizontal and vertical justification, and balancing of blocks and pages - while at the same time handling things like pickups and footnotes. Because of the "static" memory model, XPP does not waste a lot of time constantly having to recalculate "pointers" into the data as the data moves around - which would often be the case if the memory being used for this data was "dynamic".

    In any case, the "static" model for memory allocation (for these parts of a DIV's data) is what XPP is using and will be using for the foreseeable future. It is what it is.

    First, I'll note that some like to argue that (theoretically) it should not matter if the "maximum" amount of memory is allocated by a program because "it is only virtual memory and it is essentially unlimited". Well, there's theory and then there's reality. If it really didn't matter, then why is it that we all have to replace our devices every couple of years, when they become so slow that they are essentially useless to us? Because we're pretty quickly throwing more stuff at our devices (and virtually) than they can really handle.

    Now, I'll share an example showing the impact of just deferring to always using the maximum values allowed in the JT Perform Tune fields. I'll just illustrate using one of the fields, the one having the greatest impact which is the Max Page Size field.

    For example, a Max Page Size value of "99999999" equates to about 95Mb just for one internal "Page Buffer".

    The  'compose' program, and many other DIV programs, internally allocates two Page Buffers. So, that would be about 200Mb just for memory for that one XPP program for "Page Buffers" (and does not including the large chunks of memory for the list of Pages, job.dpd, the list of DLD Entries, job.dld, nor the memory needed to run the program itself).

    The 'xyview' program internally allocates four Page Buffers, so that would be about 400Mb just for memory for that one XPP program (just for Page Buffers).

    Using a customer sample that had the (unnecessarily) "maxed out" Perform Tune field values in the JT, when I ran the xyview it was a process that was 778Mb in size!

    Can you imagine how performance would be impacted by having these maxed out JT values and then having multiple users running xyview's and at the same time possibly having things like composition and printing being run via PathFinder or maybe even in Background Queues (and on the same system)? And who know what else outside of XPP might be running at the same time as well.

    The operating system is going to spend a lot of time just swapping programs in-and-out of physical memory.

    Now, if you just "randomly" cut down the Max Page Size setting to "9999999" (one less '9') - then that's a Page Buffer size of only about 10Mb. Multiply that by four (for the xyview) and you've got 40Mb - a lot less than 400Mb. Using that setting with the same customer sample, the xyview process was only 158Mb.

    And then I found that for that same customer sample (that had pretty complex and dense tables in it - one of the things that uses a lot of memory), a Max Page Size setting of "3000000" would work. This equated to a Page Buffer size of about 3Mb (and times four is 12Mb). Using that setting with that customer sample, the xyview process was only 110Mb - still "big" but a lot smaller than 778Mb.

    The 'compose' program is less than half the size of the 'xyview' and uses half as much Page Buffer memory, but the principle is the same.

    Conclusion: The best practice advice for setting the Perform Tune field values in the JT is to use values only as large as you really need to use - and don't just defer to maxing out those values.

    Jonathan Dagresta
    SDL XPP Engineering

  • Hi Bart,

    I guess I should have said on what platform I was checking and comparing program Memory usage.

    My figures were from looking at a xyview process on Solaris (Sparc) using the SZ column of the 'ps' utility - which I believe is giving a true "virtual memory" size of the process.

    I'm not sure what you were using to get your numbers on Windows, but I can see that just using Task Manager and the Memory column of the Processes tab this is not reporting a true "virtual memory" size of the (xyview) process.

    I did some web queries about Windows Task Manager and to get a "virtual memory" size of a process it appears that you have to use the Details tab and add the "Commit size" column to the display. From what I read, this column reports the true "virtual memory" size of the process/program:

    • Memory – Private Working Set – is the amount of memory used by the process which is unique to the process and cannot be accessed by other processes.
    • Memory – Commit Size – is the amount of virtual memory which is reserved, or committed, for the process.

    The "Memory (active private working set)" column, that seems to be a default column on the Details tab, is also not a true "virtual memory" size according to what I read. And it appears to be reporting a value very close to what is reported in the Memory column on the Process tab (which is the first column a Task Manager user will typically see).

    Taking that into account, on Windows I used the standard delivered "blkmerge" DIV to check things out. As delivered it has Max Page Size: xlarge, Max Pages: xlarge, and Max DLD Entries: standard.

    On the Details tab (with "Commit size" column added), Task Manager reported for xyview.exe process the following values as I tried different Max Page Size settings.

    Max Page Size:     xlarge            4000000           99999999
    Memory (a.p.w.s.): 16Mb              20Mb               123Mb
    Commit size:           50Mb             75Mb               677Mb

    So, that does illustrate (on the Windows platform) what I was showing in my initial post. It does matter what you use for the JT Perform Tune values.

    I imagine the difference between the xyview and other programs you checked (like Chrome) as to values shown in the default "Memory" columns, has to do with the "makeup" of the programs as to true "code size" versus "memory allocation", or maybe how the program allocates memory. I think that's why it's important (in this context) to make sure you are comparing the "virtual memory" size of the programs.

    Jonathan Dagresta
    SDL XPP Engineering

  • Jonathan,

    Waw, this memory thing is quite complicated in Windows.
    In the Task Manager/Details tab you can check the following numbers (yes, life is all about using the right number)
    - working set: amount of physical memory currently in use by the process
    - active private working set:  amount of physical memory currently in use by the process that can't be shared by other processes
    - shared working set: amount of physical memory  currently in use by the process that can be shared with other processes
    - commit size: amount of virtual memory reserved by the O/S for the process.

    And indeed the active private working set + shared working set = working set.
    Commit size is always a higher number.
    The shared working set is there probably because the xyview or compose is using a lot of DLLs that are/can be shared with other xyview/compose or other processes. (10 xyviews with 

    In short to me the important number to look at is active private working set as this reflects the true memory footprint/instance when running multiple instances. The commit size number is just a reservation that might or might not be in use or shared.

    For a xyview with a JT setting of 4000000 I am getting 80MB as working set/ 52MB as active private working set/ 28MB as shared working set and 93MB as commit size. 
    So 52MB is the number we need to look at.
    (same numbers for compose are 51MB/37MB/14MB/66MB)

    And this active private working set number is also the number that gets displayed in the Memory column in the default Processes tab of task manager. (I think because it is the true footprint of a process)

    Anyhow it does not really matter.
    The base rule is that the number you plug into your JT does have a big influence on the memory footprint of the xyview and compose programs. And excessive settings could lead to memory swapping and greatly reduced performance....
    So remember: Life is all about using the right number

Reply
  • Jonathan,

    Waw, this memory thing is quite complicated in Windows.
    In the Task Manager/Details tab you can check the following numbers (yes, life is all about using the right number)
    - working set: amount of physical memory currently in use by the process
    - active private working set:  amount of physical memory currently in use by the process that can't be shared by other processes
    - shared working set: amount of physical memory  currently in use by the process that can be shared with other processes
    - commit size: amount of virtual memory reserved by the O/S for the process.

    And indeed the active private working set + shared working set = working set.
    Commit size is always a higher number.
    The shared working set is there probably because the xyview or compose is using a lot of DLLs that are/can be shared with other xyview/compose or other processes. (10 xyviews with 

    In short to me the important number to look at is active private working set as this reflects the true memory footprint/instance when running multiple instances. The commit size number is just a reservation that might or might not be in use or shared.

    For a xyview with a JT setting of 4000000 I am getting 80MB as working set/ 52MB as active private working set/ 28MB as shared working set and 93MB as commit size. 
    So 52MB is the number we need to look at.
    (same numbers for compose are 51MB/37MB/14MB/66MB)

    And this active private working set number is also the number that gets displayed in the Memory column in the default Processes tab of task manager. (I think because it is the true footprint of a process)

    Anyhow it does not really matter.
    The base rule is that the number you plug into your JT does have a big influence on the memory footprint of the xyview and compose programs. And excessive settings could lead to memory swapping and greatly reduced performance....
    So remember: Life is all about using the right number

Children
No Data