Detecting language please wait for.......
I want to share some best practice advice for setting the Perform Tune values in the Job Ticket (JT), specifically with the Max Page Size, Max Pages, and Max DLD Entries fields.
I've noticed a trend in customer samples that I've seen where these field values are just "maxed out", i.e. set to their maximum values as sort of a "default" setup of the JT.
For example, I'm seeing a lot of JT's with the Max Page Size field set to 99999999 (the maximum value you can enter in that field) - even when nothing close to that value is needed for the DIV's that belong to those JOB's.
And with the same customer samples I also typically see the Max Pages and Max DLD Entries fields both set to "jumbo" (which is 1000 pages and 1500 entries), or even higher - because some users think that these are the maximum values for these fields. Note: Actually that's not the case, it's only that "jumbo" is the highest pre-defined value for these fields (the max value for these fields is actually 32767 and 262144 - but I'm not sure I should be revealing that here ).
And just using these "highest" values, by default, for these fields is not a good thing to do.
Well put simply, it will have an adverse effect on XPP performance and for no really good reason (and could adversely affect the overall performance of the system running XPP as well if other things are being done besides XPP) - hence the naming of these fields as "Perform Tune" (aka performance related).
So, what is the best practice for these fields?
Set them only as high as you need, or foresee that you will need, for the DIV's that will belong to a particular JOB.
For example (with probably the easiest of the fields to determine), if you know that no individual DIV in a JOB is going to have more than 100 pages in it then don't set the Max Pages field (needlessly) to 1000 pages (jumbo). And it may be more challenging to determine the appropriate value to use for the Max Page Size and Max DLD Entries fields, but with experience using XPP you will become pretty proficient at figuring out what values you need.
And there is rarely any lasting ill effects if you do run into the situation of having values that are too small. Yes, you'll get a "fatal" error message such as "Page Buffer Overflow …", "Page ... too large for Page Buffer", or "Number of Pages exceeds maximum ...". But then all you usually need to do is increase the appropriate field in the JT and go back and compose again.
I'm going to stop there for this initial post; many will not want to bother with the details and explanation, but for those that are interested I am going to follow with further "posts" (in this thread) to further explain and even give examples showing the impact.
Jonathan DagrestaSDL XPP Engineering
Life is all about using the right number.(a fictional story)
Today is an important day for Freddy. Freddy is a technical writer and today is D-day, D as in deliveryday. Freddy has to finish up the owner…
Jonathan - very interestingMy default settings, that I use when setting up a new job, are:- Max Page Size: 4000000 (more or less 4MB)- Max Pages: jumbo- Max DLD entries: super.On my Windows system this gives…
Jonathan,Waw, this memory thing is quite complicated in Windows.In the Task Manager/Details tab you can check the following numbers (yes, life is all about using the right number)- working set: amount…
Here are more details and an example illustrating the impact of all of this.
In general there are two "philosophies" in memory allocation for a program's use when dealing with data - "static" versus "dynamic". A "static" approach means that memory is allocated up front (and does not move) and a "dynamic" approach means that memory is allocated (and reallocated) as needed. But typically when memory is reallocated (to a bigger size), that also means that the location of that memory (within a program's address space) has moved to a new location.
For some of the larger chunks of data associated with a DIV, from the beginning XPP has used a "static" type of memory allocation. These include the memory for "Page Buffers" (used to hold the contents of pages - Max Page Size), the memory used to hold the list of pages (corresponding to the job.dpd file - Max Pages), and the memory used to hold the list of "named" items such as footnotes and pickups (corresponding to the job.dld file - Max DLD Entries).
There are a number of reasons for having done this, and although some of those reasons might not be as important as they were at the "beginning" of XPP there are still some good reasons for keeping this approach. A big one is still the speed with which XPP is able to do very complicated composition calculations including things like laying out pages, horizontal and vertical justification, and balancing of blocks and pages - while at the same time handling things like pickups and footnotes. Because of the "static" memory model, XPP does not waste a lot of time constantly having to recalculate "pointers" into the data as the data moves around - which would often be the case if the memory being used for this data was "dynamic".
In any case, the "static" model for memory allocation (for these parts of a DIV's data) is what XPP is using and will be using for the foreseeable future. It is what it is.
First, I'll note that some like to argue that (theoretically) it should not matter if the "maximum" amount of memory is allocated by a program because "it is only virtual memory and it is essentially unlimited". Well, there's theory and then there's reality. If it really didn't matter, then why is it that we all have to replace our devices every couple of years, when they become so slow that they are essentially useless to us? Because we're pretty quickly throwing more stuff at our devices (and virtually) than they can really handle.
Now, I'll share an example showing the impact of just deferring to always using the maximum values allowed in the JT Perform Tune fields. I'll just illustrate using one of the fields, the one having the greatest impact which is the Max Page Size field.
For example, a Max Page Size value of "99999999" equates to about 95Mb just for one internal "Page Buffer".
The 'compose' program, and many other DIV programs, internally allocates two Page Buffers. So, that would be about 200Mb just for memory for that one XPP program for "Page Buffers" (and does not including the large chunks of memory for the list of Pages, job.dpd, the list of DLD Entries, job.dld, nor the memory needed to run the program itself).
The 'xyview' program internally allocates four Page Buffers, so that would be about 400Mb just for memory for that one XPP program (just for Page Buffers).
Using a customer sample that had the (unnecessarily) "maxed out" Perform Tune field values in the JT, when I ran the xyview it was a process that was 778Mb in size!
Can you imagine how performance would be impacted by having these maxed out JT values and then having multiple users running xyview's and at the same time possibly having things like composition and printing being run via PathFinder or maybe even in Background Queues (and on the same system)? And who know what else outside of XPP might be running at the same time as well.
The operating system is going to spend a lot of time just swapping programs in-and-out of physical memory.
Now, if you just "randomly" cut down the Max Page Size setting to "9999999" (one less '9') - then that's a Page Buffer size of only about 10Mb. Multiply that by four (for the xyview) and you've got 40Mb - a lot less than 400Mb. Using that setting with the same customer sample, the xyview process was only 158Mb.
And then I found that for that same customer sample (that had pretty complex and dense tables in it - one of the things that uses a lot of memory), a Max Page Size setting of "3000000" would work. This equated to a Page Buffer size of about 3Mb (and times four is 12Mb). Using that setting with that customer sample, the xyview process was only 110Mb - still "big" but a lot smaller than 778Mb.
The 'compose' program is less than half the size of the 'xyview' and uses half as much Page Buffer memory, but the principle is the same.
Conclusion: The best practice advice for setting the Perform Tune field values in the JT is to use values only as large as you really need to use - and don't just defer to maxing out those values.
Jonathan - very interestingMy default settings, that I use when setting up a new job, are:- Max Page Size: 4000000 (more or less 4MB)- Max Pages: jumbo- Max DLD entries: super.On my Windows system this gives me for a typical job a xyview with a footprint of 50-55MB and a compose 35-38MB.To put things in perspective here are some other windows program numbers:- Chrome - 1 active tab: 500MB- Firefox - 1 active tab: 350MB- the default windows antivirus 'Antimalware Service Executable': 180MB- eM Client (my mail client): 100MB - dropbox: 67MB- windows explorer (which I never use, so with no open windows): 35MB- windows task manager: 35MB
I guess I should have said on what platform I was checking and comparing program Memory usage.
My figures were from looking at a xyview process on Solaris (Sparc) using the SZ column of the 'ps' utility - which I believe is giving a true "virtual memory" size of the process.
I'm not sure what you were using to get your numbers on Windows, but I can see that just using Task Manager and the Memory column of the Processes tab this is not reporting a true "virtual memory" size of the (xyview) process.
I did some web queries about Windows Task Manager and to get a "virtual memory" size of a process it appears that you have to use the Details tab and add the "Commit size" column to the display. From what I read, this column reports the true "virtual memory" size of the process/program:
The "Memory (active private working set)" column, that seems to be a default column on the Details tab, is also not a true "virtual memory" size according to what I read. And it appears to be reporting a value very close to what is reported in the Memory column on the Process tab (which is the first column a Task Manager user will typically see).
Taking that into account, on Windows I used the standard delivered "blkmerge" DIV to check things out. As delivered it has Max Page Size: xlarge, Max Pages: xlarge, and Max DLD Entries: standard.
On the Details tab (with "Commit size" column added), Task Manager reported for xyview.exe process the following values as I tried different Max Page Size settings.
Max Page Size: xlarge 4000000 99999999Memory (a.p.w.s.): 16Mb 20Mb 123MbCommit size: 50Mb 75Mb 677Mb
So, that does illustrate (on the Windows platform) what I was showing in my initial post. It does matter what you use for the JT Perform Tune values.
I imagine the difference between the xyview and other programs you checked (like Chrome) as to values shown in the default "Memory" columns, has to do with the "makeup" of the programs as to true "code size" versus "memory allocation", or maybe how the program allocates memory. I think that's why it's important (in this context) to make sure you are comparing the "virtual memory" size of the programs.
Today is an important day for Freddy. Freddy is a technical writer and today is D-day, D as in deliveryday. Freddy has to finish up the owner manuals for the new Toyota that will be launched Europe wide next month. And for this to happen, the PDFs have to be delivered at the printer by tomorrow morning at the latest.
Today is also an important day for Freddy for another reason. This is his 15th wedding anniversary and his marriage needs a little boost. And Freddy has a plan, Freddy will jump into the local Victoria's Secret and get what he think will please his wife. As the store closes by 6, Freddy needs to leave the office by 5 at the very latest.
This morning the Toyoata marketing team finally decided on the name of the new motor. It will no longer be called the UHF-1000. No, it will called by its full name: Ultra Hybrid Fuel-1000. Freddy plugs in this new name into his SDL Tridion Docs system and launches a last proof of the 350 page english version of the owners manual. While waiting for the system, Freddy makes a call to Anne, his wife's best friend and gets confirmation for the correct numbers for his present: 95C. It is getting late and Freddy really wants to go leave the office, so he anxiously awaits the PDF. 10 minutes later, disaster strikes. The longer name for the new motor makes the motor spec table to overflow the page and that is a no-no. Freddy decides to plug in a smaller point size into the motor spec table. There is no more time for yet another proof so Freddy launches the complete batch of the 22 different language versions. Freddy happily leaves the office at 5 minutes past 5 and runs on his way to the train station in order to make up for the lost time. He just makes the train and can now relax. Mission completed as tomorrow morning he will find the 22 PDFs on his desk and send them to the printer. And more importantly he now has the time to get his special present for his wife, so tonight will be a success
The next day
Freddy knocks in full panic mode on the door of John's office. This morning he found 0 PDFs and he knows he will never be able to deliver the 22 PDFs in time. He quickly calculates that 22 times 10 minutes is 220 minutes or give or take 4 hours. "What's wrong, Freddy? You look like you had a rough night" John, the XPP system manager, asks. "John, I needed the PDFs by 9 this morning, but nothing came out of the system and I can't figure out what went wrong. Yesterday it all worked fine..." "So what did you change last?" "Well I reduced the point size on the motor spec table page, could it be that? Don't tell me this is that same problem we had 6 months ago?" "Ah the Max Page Size setting? Could very well be, lets up the Max Page Size again and give it a try" "But didn't you do that same thing like 6 months ago" Freddy asks. "Well apparently I did not plug in a high enough number, it is all about using the right number" says John. Ten minutes later the PDF was there. "Let's quickly queue up the other 21 languages" says a relieved Freddy.
Needless to say that Freddy did not make his deadline and there was a lot hassle with Toyota because of that. So much that Peter, the CEO of the service company that Freddy and John were working for, got involved. Peter promised Toyota that he would go to the bottom of this problem. So he sets up emergency meeting John and Freddy. "Our customer did not like this at all. So tell me guys what was going on here? How is this even possible?" utters an annoyed Peter. "Well it is all about getting the right number into the XPP system" John tries. "I don't care about the 'right' number. We had exactly the same problem six months ago and now again. Tell me how is this possible?" shouts Peter. John and Freddy both keep quiet. "What is the biggest number you can put into the system?" says Peter. "9999999" says John quietly. "Well then use" shouts Peter, signaling that the meeting is over.
So be it. John reluctantly plugs in the magic number 9999999 into the Max Page Size field.
Since then, Freddy never missed a deadline again, John kept his job and Peter kept relatively happy.
Life is all about the right number
PS: many years later, John opens up a ticket with SDL support and sends them a sample job. SDL support comes back to him with the suggestion to lower the Max Page Size setting. But John wants to keep his job and does not change the Max Page Size setting!
Jonathan,Waw, this memory thing is quite complicated in Windows.In the Task Manager/Details tab you can check the following numbers (yes, life is all about using the right number)- working set: amount of physical memory currently in use by the process- active private working set: amount of physical memory currently in use by the process that can't be shared by other processes- shared working set: amount of physical memory currently in use by the process that can be shared with other processes- commit size: amount of virtual memory reserved by the O/S for the process.And indeed the active private working set + shared working set = working set.Commit size is always a higher number.The shared working set is there probably because the xyview or compose is using a lot of DLLs that are/can be shared with other xyview/compose or other processes. (10 xyviews with In short to me the important number to look at is active private working set as this reflects the true memory footprint/instance when running multiple instances. The commit size number is just a reservation that might or might not be in use or shared.For a xyview with a JT setting of 4000000 I am getting 80MB as working set/ 52MB as active private working set/ 28MB as shared working set and 93MB as commit size. So 52MB is the number we need to look at.(same numbers for compose are 51MB/37MB/14MB/66MB)And this active private working set number is also the number that gets displayed in the Memory column in the default Processes tab of task manager. (I think because it is the true footprint of a process)Anyhow it does not really matter.The base rule is that the number you plug into your JT does have a big influence on the memory footprint of the xyview and compose programs. And excessive settings could lead to memory swapping and greatly reduced performance.... So remember: Life is all about using the right number