-
Notifications
You must be signed in to change notification settings - Fork 15
Require list expansion
Require list expansion is the expansion of the dependency list specified in a require call to include implied, nested, dependencies (i.e. the list of modules specified in the define() methods of the modules in the require list, as well as their dependencies, and so on). The goal of require list expansion is to reduce the number of HTTP requests needed to load AMD modules by avoiding the cascade of requests caused by dependency discovery as new modules are loaded. By expanding the require list to include all nested dependencies, all of the modules needed by a given require call can be requested from the Aggregator using a single request/response.
Require list expansion is optional, and is enabled by setting the expandRequire property in the loader extension config.
Require list expansion is one way of reducing the number of HTTP requests needed to load a set of modules and its dependencies. Server-side layer expansion is another way. The Aggregator supports both. See Require list expansion vs. Server-side layer expansion for a discussion about the advantages and disadvantages of each approach. Require list expansion and Server-side layer expansion are mutually exclusive. If both enabled, then Server-side layer expansion will prevail.
Require list expansion is done by the Aggregator as part of the JavaScript minification process by a custom compiler pass module. The parsed AST (Abstract Syntax Tree) is scanned for require calls, and for each one found, the require list specified in the call is expanded to include nested required modules. So given the following require call:
require(["moduleA"], function(moduleA) {
// Print a message to the console
console.log("moduleA loaded");
});
and assuming that moduleA lists in it's define() function that it depends on moduleB, and moduleB lists in it's define() function that it depends on moduleC, then the resulting JavaScript output by the Aggregator will look like the following:
require(["moduleA","moduleB","moduleC"],function(moduleA){console.log("moduleA loaded");});
Note that expanded dependencies are always specified following the initial, explicit, require list which is not modified by the Aggregator. Only dependencies which are specified as a string literal within an anonymous array declared inside the require call are expanded. Anything else (i.e. an array reference or a reference to a string within the anonymous array) are left as is.
Require list expansion can also be performed for the deps property in the client-side AMD loader config. The deps property must be assigned an array literal, and must be a property of an object called require, or else an object who's name is specified by the configVarName URL parameter. The following forms of deps property assignment are recognized by the Aggregator for the purpose of expanding the specified dependency list:
require.deps = [...];
require = { deps: [...] };
var require = { deps: [...] };
require: { deps: [...] }};
obj.require = { deps: [...] };
In order to minimize the impact of require list expansion on code size, expanded require lists are trimmed to exclude any modules contained in the following:
- All of the modules in the response containing the module being expanded, as well as their nested dependencies.
- The modules included in the application's bootstrap layer using the
depsandpreloadsURL parameters, and their expanded dependencies. - Any modules specified in the dependency lists associated with the closures created by enclosing define() and require() function callbacks, including all of those modules nested dependencies.
The Aggregator provides support for the has! loader plugin when used in specifying dependent modules in an AMD require or define call. The has! loader plugin enables conditional loading of dependent modules based on has.js feature detection, meaning that the loader will load the dependent modules only if the specified feature is supported. An example of a define call using the has! loader plugin is:
define(["dojo", "dojo/has!host-browser?dojo/domReady", "dojo/has"], function(dojo, domReady) {
In this example, the domReady module will be loaded only if the host-browser feature is supported.
When expanding the dependency list of a require call who's dependencies include a module specified using the has! loader plugin, the Aggregator will include that module's dependencies in the set of expanded dependencies only if the feature specified by the plugin is supported as determined by the feature set provided in the request. An important caveat is that the feature set used when the module dependencies are expanded is not necessarily the same as the feature set in effect when the require call is executed. The code to add the test for the feature may not have run yet when the Aggregator tries to expand the module's dependencies, even though it will have been added by the time the require function is called. If a feature specified using the has! loader plugin is not defined at the time that the require list is being expanded, then the aggregator will perform has! plugin branching (see below) to expand both branches of the expression, or if has! plugin branching is disabled by the disableHasPluginBranching option, then no dependency expansion of the modules in the expression will occur.
The Aggregator recognizes has! loader plugins specified using any path, so has!, dojo/has!, foo/has!, etc. will all be recognized as the has! loader plugin and treated the same for the purposes of require list expansion.
When a feature specified in a has! plugin expression is not defined at the time that require list expansion is performed, the aggregator will expand both branches of the conditional, conditionalizing the expanded dependencies for each branch using the has! loader plugin. Consider the following require call:
require(["has!foo?foo:bar"],
If the feature "foo" is defined when the aggregator performs require list expansion on this require call, then the require list will be expanded to include the expanded dependencies for either the module foo, or the module bar, depending upon the value of the feature. If the feature "foo" is not defined, then the aggregator will include in the require list the expanded dependencies for both modules, prefixing each expanded dependency with the has! loader plugin conditional for the expanded branch. For example, if the define function for the module foo is as follows:
define(["foodep1", "foodep2"],
and the define function for the module bar is as follows:
define(["bardep1", "bardep2"],
then the expanded dependencies for the require call above would be as follows:
require(["has!foo?foo:bar", "has!foo?foodep1", "has!foo?foodep2", "has!foo?:bardep1", "has!foo?:bardep2"],
Of course, the branched dependencies may themselves use the has! loader plugin for the same or other features, so the aggregator will combine the has! loader plugin expressions into nested has conditionals, detecting merged branches, redundant conditionals and mutually exclusive conditionals, and optimizing the plugin expressions or eliminating the dependency as appropriate. The aggregator will also detect that if any module is unconditionally part of the expanded dependency list, then any has! loader plugin expressons in the expanded dependencies that conditionally require the module may be omitted.
In order to perform require list explosion, the Aggregator builds a dependency graph of all of the AMD modules contained within the resource bundles and directories specified by the server-side AMD configuration. This involves opening each javascript file and parsing the javascript to locate and identify the module's dependencies as specified in the define() for the module. This is somewhat time consuming and can over a minute the first time the servlet is initialized. The dependency graph is serialized to disk and on subsequent server restarts, the serialized dependency graph is de-serialized and re-used. Because of the poor performance of file last-modified checking in Java, the de-serialized dependency graph is NOT validated against the last-modified times of the source files on every server restart unless specifically enabled Aggregator options setting. Java 7 introduces the java.nio.file package, which provides facilities for efficiently traversing a directory tree and querying directory entry meta-data, including file last-modified times. It is anticipated that the performance improvements realized by using these facilities will enable dependency graph validation at servlet initialization within a reasonable amount of time, and will be enabled by default.
Validation of dependencies is performed at servlet initialization if the verifyDeps option is enabled. In any case, you can validate dependencies after making a change to source files that affect dependencies using the validateDeps console command:
aggregator validateDeps <servlet-name> [clean]
where the optional clean argument tells the Aggregator to discard its cached dependency list and rebuild it from scratch, re-parsing all AMD modules regardless of whether the file has changed since it was last parsed for dependencies.
Changes to server-side AMD config files can similarly affect dependencies. Because config file changes require the dependency graph to be re-validated, these changes also will NOT be detected on server restart unless enabled by properties settings as indicated above. When making changes to config files, run the console command
aggregator reloadconfig <servlet-name>
in order to reload the config file, re-validate the dependencies and realize the effect of these changes in the Aggregator.
The above console commands are provided for use in a development environment where application files are being modified and you want to see the results of those modifications. In a production environment, it should not be necessary to use these commands. When upgrading a deployed application, the install or update program should clear the Aggregator caches by deleting the cache directories used by the Aggregator as described below.
Require list expansion has important caching implications for developers. Because changes to the require list in a module's define() function can have ripple effects that extend to other cached responses in ways that are difficult to evaluate efficiently, the Aggregator does not attempt to detect these changes. Instead, when validating dependencies, if a module's required dependencies are determined to have changed, then the entire cache for that servlet is invalidated. This is the equivalent of executing a clearcache console command for the affected servlet.
Because re-validation of the dependency graph is only performed at server restart (if enabled by option setting) or in response to a console command, this means that if you edit a module's dependencies and save those changes, then any responses from the Aggregator that include the changed module will have that module's changes (as long as the developmentMode option is enabled), but any expanded require lists in other modules that include the changed module's dependencies will be stale until dependencies have been re-validated.
Note that using the clearcache console command is not sufficient to rebuild the expanded require lists because this will just cause the new responses to be built using the stale dependency graph. The dependency graph will not be rebuilt until dependencies are re-validated, however, re-validation does perform the equivalent of clearcache console command if dependencies have changes.
Another way to realize dependency graph changes is to delete the contents of cache directories, which deletes the cached responses as well as the serialized dependency graph. This technique might be utilized by installation or upgrade scripts, for example. The cache directories are located in the state location for the servlet plugin on the server's file system (the value returned by Plugin.getStateLocation()). Each servlet has a directory in this location who's name is derived from the servlet's name (the value of the alias attribute in the element in the plugin configuration for the servlet).
Require list expansion takes place only in JavaScript files, not in script code within HTML pages. Consequently, require calls executed from HTML pages (within an onClick handler, etc.) can result in the cascade of HTTP requests for required modules resulting from dependency discovery as modules are loaded. To avoid this, use a wrapper function within the module for the page or widget that needs to require additional modules. For example, instead of having:
<button name="getNames" onclick="require(["dialogs/getNames"], function(dlg) {dlg.show();});">
Require the module for the page or widget that is displaying the button (it's already loaded, so the require call will complete synchronously), and within the completion function, call the wrapper function to load the getNames dialog:
<button name="getNames" onclick="require(["scenes/mailScene"], function(mail) {mail.getNamesDlg(function(dlg){dlg.show();}});">
In this case, getNamesDlg is a simple wrapper method in the scenes/mailScene module (the module for the widget that contains the button) that calls require:
getNamesDlg: function(callback) {
require(["dialogs/getName"], callback);
}
Because getNamdeDlg is in a JavaScript module, the dependency list in the require call will be expanded to include all of the nested dependencies (excluding the declared and nested dependencies of any containing closures and the require.deps array), eliminating the need for extra HTTP requests resulting from dependency discovery.
Require list expansion necessarily increases the size of the JavaScript code. While much of the resulting increase is in the form of repeated patterns that will compress nicely when the response is gzip encoded, it is still desirable to limit the expansion to the extent possible. This is done by trimmming expanded require lists to exclude modules which are known to already be loaded. One way that the Aggregator has of knowing which modules have already been loaded is by analyzing the dependency lists of enclosing require() or define() closures. In order to take full advantage of this, it is necessary to ensure that the dependency lists of enclosing define() functions are complete. The dependency lists of define functions are not expanded in the JavaScript code, yet the expanded dependency lists of define functions are excluded from the expanded dependencies of any nested require calls.
For example, if a module requires dijit, but it is known that the dijit sub-system is already loaded, you might choose to omit dijit from the dependency list in the define function for the module and use the global dijit reference to access dijit methods:
define([], function() {
getOrCreateFoo: function(callback) {
var foo = dijit.byId("foo");
return (foo) ? callback(foo) : require(["Foo"], function(Foo) {callback(new Foo("foo"));});
}
});
This will work, but should be avoided for several of reasons. First, it makes your code less portable. Avoiding access to global objects potentially allows multiple applications, each using the version of dojo specified in their loader config, to co-exist peacefully on the same page. Secondly, the names of local and function variables can be renamed by the javascript optimizer to make them smaller, while the names of globals cannot. And finally, the expanded require list will be larger than it needs to be because the list may need to include all of dijit's expanded dependencies (assuming that Foo depends directly or indirectly on dijit). To avoid all these issues, the above example should instead be written as:
define(["require", "dijit"], function(require, dijit) {
getOrCreateFoo: function(callback) {
var foo = dijit.byId("foo");
return (foo) ? callback(foo) : require(["Foo"], function(Foo) {callback(new Foo("foo"));});
}
});
In this example, the expanded dependency list for the require call will be smaller because is will exclude all of dijit's declared and nested dependencies.
In version 1.2 and greater, the expanded require list is no longer added to the array literal in the require call. Instead, the expanded require list is declared in an array variable which is appended to the application specified dependencies using the JavaScript Array.concat() function. This is done in support of module name id mapping of requested modules, which allows the aggregator to specify the names of requested modules on the URL in a much more compact manner, reducing the need for request splitting when URL length limits are exceeded.
Using module name id mapping, the aggregator associates the module names in expanded require lists with module ids (numbers) which are used instead of the names when the loader requests modules from the aggregator.
The Aggregator provides the following tools for analyzing require list expansion:
-
getdepsconsole command - this command can be used to determine the expanded dependencies of any module. -
expandRequiresloader extension config param - When set to the valuelog, enables logging of require list expansion details to the browser console usingconsole.log(). This information can be used to help determine how require lists are expanded, and why individual modules are included, or not include, in the expansion.