Implemented parallel building of multiple tools, and documented it.

git-svn-id: svn+ssh://svn.gna.org/svn/gnustep/tools/make/trunk@29542 72102866-910b-0410-8b05-ffd578937521

ChangeLog

@@ -1,3 +1,21 @@
+2010-02-11  Nicola Pero  <nicola.pero@meta-innovation.com>
+
+	Implemented experimental support for parallelizing building
+	different tools.  For example, if you have "TOOL_NAME = autogsdoc
+	cvtenc", the two tools are now built in parallel when you specify
+	'make -j 2'.  Warning: this means the order of building tools is
+	no longer guaranteed.
+	* Master/tool.make (internal-all): When parallel building is
+	enabled, fire off a _GNUSTEP_MAKE_PARALLEL=yes sub-make to build.
+	(internal-master-tool-all): New target used by the
+	_GNUSTEP_MAKE_PARALLEL=yes sub-make when parallel building is
+	enabled.
+	* Instance/rules.make (OBJ_DIRS_TO_CREATE): When building in
+	parallel, ignore failures to create an object subdirectory.
+	* Documentation/releasenotes.texi: Updated.
+	* Documentation/RELEASENOTES: Regenerated.
+	* RELEASENOTES: Updated.
+	
 2010-02-07  Nicola Pero  <nicola.pero@meta-innovation.com>
 
 	Added support for having header files in sub-subdirectories.  Just

Documentation/releasenotes.texi

@@ -7,6 +7,31 @@ migrate to using a newer version of the make system.
 @section Version 2.2.1
 @table @samp
 
+@item the order in which tools are built is no longer guaranteed
+If you use something like ``TOOL_NAME = ToolA ToolB'' to build
+multiple tools in the same GNUmakefile, you need to be aware that the
+way the tools are built changed in version 2.2.1.  Before version
+2.2.1, the tools were always built one after the other one, exactly in
+the order specified.  So, in the example ToolA would be built before
+ToolB.  Starting with 2.2.1, the tools might be built completely in
+parallel if parallel building is enabled.  So, the order in which they
+are built is no longer defined and your GNUmakefile should not depend
+on the order in which tools are specified in the GNUmakefile.  Most
+GNUmakefiles should be unaffected because rarely people rely on the
+order in which tools are built.  If your GNUmakefile does depend on
+the order in which tools are built, you have a few options.  The
+preferred option is to identify the code or steps that need to be
+executed before some of the tools are built (for example, you may be
+generating a customized header in before-ToolA-all::, but the header
+is actually used when building not only ToolA, but ToolB as well, so
+the build breaks if ToolB is not built after ToolA) and put them into
+a before-all:: rule, which is guaranteed to be executed before
+everything else.  In this way your serialized code is executed first,
+and the build can continue in a complete parallel fashion afterwards.
+If all else fails, you can still disable parallel building in your
+GNUmakefile by adding GNUSTEP_MAKE_PARALLEL_BUILDING=no just after
+including common.make.
+
 @item support for having source files in subdirectories
 Starting with version 2.2.1, it is possible to put source files in
 subdirectories by specifiying them as in xxx_OBJC_FILES =

Instance/rules.make

@@ -262,8 +262,32 @@ OBJ_FILES_TO_LINK = $(strip $(C_OBJ_FILES) $(OBJC_OBJ_FILES) $(CC_OBJ_FILES) $(O
 # covered by default.
 OBJ_DIRS_TO_CREATE = $(filter-out $(GNUSTEP_OBJ_DIR)/,$(sort $(dir $(OBJ_FILES_TO_LINK))))
 
+ifeq ($(GNUSTEP_MAKE_PARALLEL_BUILDING), no)
 $(OBJ_DIRS_TO_CREATE):
-	$(ECHO_CREATING)cd $(GNUSTEP_BUILD_DIR); $(MKDIRS) $@$(END_ECHO)	
+	$(ECHO_CREATING)cd $(GNUSTEP_BUILD_DIR); $(MKDIRS) $@$(END_ECHO)
+else
+# When doing a parallel build, we build instances in parallel.  If two
+# instances need to create the same directory, there could be a race
+# condition and one of the two might fail.  So we use '-' here to tell
+# GNU make to ignore any such errors and keep going.  If the error was
+# a legitimate one (not a concurrency-related one) the build will fail
+# later when the object file can't be put into the directory.  PS:
+# This all can be avoided if you avoid having source files for two
+# instances in the same subdirectory.
+#
+# FIXME/TODO: The better solution would be to prefix the
+# OBJ_DIRS_TO_CREATE with the instance name to avoid any clashes.
+# Then each instance has its own completely separate ./obj/ToolName/
+# directory for its object files and there can be no concurrency
+# issues.  Unfortunately, this requires changing the top-level rules,
+# and would break the "API" as the location of object files would
+# change.  Any GNUmakefile using $(GNUSTEP_OBJ_DIR) might potentially
+# be broken.  So it's a major change and we leave it for 2.4.0.  For
+# now, the hack of ignoring concurrency errors in the unlikely case
+# that two instances need the same OBJ_DIRS_TO_CREATE is enough.
+$(OBJ_DIRS_TO_CREATE):
+	-$(ECHO_CREATING)cd $(GNUSTEP_BUILD_DIR); $(MKDIRS) $@$(END_ECHO)
+endif
 
 # If C++ or ObjC++ are involved, we use the C++ compiler instead of
 # the C/ObjC one to link; this happens automatically when compiling

Master/tool.make

@@ -26,8 +26,44 @@ ifeq ($(RULES_MAKE_LOADED),)
 include $(GNUSTEP_MAKEFILES)/rules.make
 endif
 
+ifeq ($(GNUSTEP_MAKE_PARALLEL_BUILDING), no)
+
+# Standard building
 internal-all:: $(TOOL_NAME:=.all.tool.variables)
 
+else
+
+# Parallel building.  The actual compilation is delegated to a
+# sub-make invocation where _GNUSTEP_MAKE_PARALLEL is set to yes.
+# That sub-make invocation will fire off the building of the tools in
+# parallel.  This is great as the entire building (including the
+# linking) of the tools is then parallelized.
+
+# Please note that we need to create the ./obj directory before we
+# fire off all the parallel sub-makes, else they'll be a race
+# condition to create it. (typically what happens is that two
+# sub-makes detect that it needs creating, the first one creates it,
+# and when the second one tries to create it, it will fail as it's
+# already been created).
+internal-all:: $(GNUSTEP_OBJ_DIR)
+	$(ECHO_NOTHING)$(MAKE) -f $(MAKEFILE_NAME) --no-print-directory --no-keep-going \
+	internal-master-tool-all \
+	GNUSTEP_BUILD_DIR="$(GNUSTEP_BUILD_DIR)" \
+	_GNUSTEP_MAKE_PARALLEL=yes$(END_ECHO)
+
+internal-master-tool-all: $(TOOL_NAME:=.all.tool.variables)
+
+endif
+
+# TODO: Installing and uninstalling in parallel would be extremely
+# cool, but if you fire off many sub-makes (one for each instance) in
+# parallel, you end up with a lot of race conditions as the tools are
+# most often installed in the same directories, which the different
+# sub-makes will attempt to create concurrently.  A better solution
+# would be to fire off a single Master invocation with
+# _GNUSTEP_MAKE_PARELLEL enabled, and in there install all the tools
+# using parallel rules.  This requires moving all the tool installation 
+# code from Instance/ to Master/.
 internal-install:: $(TOOL_NAME:=.install.tool.variables)
 
 internal-uninstall:: $(TOOL_NAME:=.uninstall.tool.variables)
@@ -52,6 +88,8 @@ internal-clean::
 	rm -rf $(GNUSTEP_BUILD_DIR)/Resources
 endif
 
+# TODO: It should be really safe to parallelize the 'strings' targets,
+# but it's worth checking to make sure we're not breaking anything.
 internal-strings:: $(TOOL_NAME:=.strings.tool.variables)
 
 $(TOOL_NAME):

RELEASENOTES

@@ -8,6 +8,32 @@ using a newer version of the make system.
 1.1 Version 2.2.1
 =================
 
+`the order in which tools are built is no longer guaranteed'
+     If you use something like "TOOL_NAME = ToolA ToolB" to build
+     multiple tools in the same GNUmakefile, you need to be aware that
+     the way the tools are built changed in version 2.2.1.  Before
+     version 2.2.1, the tools were always built one after the other
+     one, exactly in the order specified.  So, in the example ToolA
+     would be built before ToolB.  Starting with 2.2.1, the tools might
+     be built completely in parallel if parallel building is enabled.
+     So, the order in which they are built is no longer defined and
+     your GNUmakefile should not depend on the order in which tools are
+     specified in the GNUmakefile.  Most GNUmakefiles should be
+     unaffected because rarely people rely on the order in which tools
+     are built.  If your GNUmakefile does depend on the order in which
+     tools are built, you have a few options.  The preferred option is
+     to identify the code or steps that need to be executed before some
+     of the tools are built (for example, you may be generating a
+     customized header in before-ToolA-all::, but the header is
+     actually used when building not only ToolA, but ToolB as well, so
+     the build breaks if ToolB is not built after ToolA) and put them
+     into a before-all:: rule, which is guaranteed to be executed before
+     everything else.  In this way your serialized code is executed
+     first, and the build can continue in a complete parallel fashion
+     afterwards.  If all else fails, you can still disable parallel
+     building in your GNUmakefile by adding
+     GNUSTEP_MAKE_PARALLEL_BUILDING=no just after including common.make.
+
 `support for having source files in subdirectories'
      Starting with version 2.2.1, it is possible to put source files in
      subdirectories by specifiying them as in xxx_OBJC_FILES =