Training a network involves many choices, and the opportunity to learn from past mistakes.    Criteria for a successfully trained network are:

The points in the left picture should lie on the unit circle.

The line segments in the right picture should be close to radial (denoted by red color).

Network 1: The point cloud is close to the unit circle only in the first quadrant, and it is diffuse.  The lines are mostly radial only in the first quadrant, and chaotic.  (not (HelmetsTrain6.nb, “level5round1.wlnet”)

Network 1a is a fine-tuning of network 1, transferring the previous learning and training more layers.    The point cloud is less diffuse (good), but the overall fit to the circle is not better (bad).   The line segments are only radial in the second quadrant, but less chaotic.  (HelmetsTrain6.nb,“level5round2.wlnet”)

Network 2 is designed to constrain the point to the unit circle (think Lagrange multipliers).   The points are not close to the unit circle, but they are close to a smaller circle.  The line segments are chaotic, but the pattern is uniform. (from HelmetsTrain8, "constrainedMacNet.wlnet")

Network 2a is a fine-tuning of Network 1 using transfer of learning from network 1.  The modification is  sharper constraint enforcement.  This is something of an improvement, but not uniform across the circle.  (from HelmetsTrain8, “tunedSharpNet.wlnet”)

Network 2b is a fine-tuning of Network 2a, transferring learning and training more layers.  This is an improvement.  The points are uniformly close to the circle, and the line segments are largely radial and nonchaotic.   But there are regions where there is a systematic error (denoted by green).  (from HelmetsTrain8, “tunedSharpNet.wlnet”)

The main result was the largely successful incremental training of  large convolutional network to identify nonplanar rotations of single three-dimensional object. For simplicity of diagnosing and tuning the results, the rotations were limited to rotations about the vertical axis in the image plane.  This case is simpler since the restricted rotation group SO(2) can be described as a circle in two-dimensions, which is easily visualized.   The full rotation group is higher-dimensional, making analogous visualizations of results more difficult.    The results obtained for the restricted rotation group SO(2) should generalize to allow for arbitrary three-dimensional rotations in SO(3) of a single object.  

The key feature was the introduction of  a network layer to dynamically enforce the constraint that a sequence of numbers satisfy a (nonlinear) constraint.    This made it possible to incrementally transfer learning and tune the network with good results.   In contrast, network architectures that did not enforce this constraint on the output gave results that did not satisfy the constraint, and were therefore a poor fit.

Link to the GitHub

Explicit code

Generation of initial training/validation/test data (Helmets6.nb). Objecdt is rotated about the vertical axis in the image plane, and illuminated by with three random colors each from a separate random direction in the front hemisphere.  It takes about a second to generate one element of the training set, so only a few thousand elements can be generated within the time constraints.    The image is generated along with the {Cos,Sin} of the rotation angle.

Mesh of 3D object (.stl format, downloaded from thingiverse.com)

iron =;

helmetType=iron;

random unit vector

randomUnitVector := RandomVariate[CircularRealMatrixDistribution[3]].{1,0,0}

random vector in front hemisphere, used for lighting directions to avoid lighting from the back

frontVec[refVec_] := Module[{tempVec =randomUnitVector},If[tempVec.refVec ≥ 0,tempVec,-tempVec]]

Generating a training set

helmetName = "iron"
helmetType = ToExpression[helmetName];
setIndex = 0
sampsize =5
imsize=64;

ClearAll[temp]
temp =Table[ranAng=RandomReal[2 Pi];
cs=Cos[ranAng];
ss=Sin[ranAng];
color1=Hue[RandomReal[1]];
color2=Hue[RandomReal[1]];
color3=Hue[RandomReal[1]];
rotMat={{cs,ss,0},{-ss,cs,0},{0,0,1}};
 viewVec=rotMat.{0,-1.5,0};
s=Show[helmetType,ViewPoint->viewVec,Lighting->{
{"Directional",color1,{frontVec[viewVec],{0,0,0}}},{"Directional",color2,{frontVec[viewVec],{0,0,0}}},{"Directional",color3,{frontVec[viewVec],{0,0,0}}}
},ImageSize->imsize];
im=Rasterize[s,"Image",ImageSize-> {imsize,imsize}];
im -> {cs,ss}
,{nls,1,sampsize}]

Export[StringJoin[helmetName,"Helm",ToString[setIndex],".m"],temp]

Data augmentation generates more training set elements from the initial training set, more quickly than generating new images.    This allows generation of a large training set ( 152960 elements), validation set ( 38240 element), and test set (about 1000 elements) within time constraints.

Data Augmentation by reflection (angle must also be flipped)

keysReflect[image_] := ImageReflect[image,Left-> Right]

valuesReflect[twovec_] := {First[twovec],-Last[twovec]}

ClearAll[reflectRule]
reflectRule[rule_] := keysReflect[Keys[rule]]-> valuesReflect[Values[rule]]
SetAttributes[reflectRule,Listable]

reflectRule[temp]

Data Augmentation by shifts (small shifts such as 1 or 2 pixels are used, a large shift is used below for demonstration purposes)

ClearAll[verticalShift]
verticalShift[image_,range_] := Module[{temp,shift=RandomInteger[{1,Abs[range]}],sign=1-2*RandomInteger[1]},
temp=ImageTake[image,Sign[sign]*(64-shift)];
ImagePad[temp,{{0,0},{(1/2)*(-sign+1)shift,(1/2)*(sign+1)shift}},White]
]

clockwise[image_] :=ImageRotate[image,Top->Right]
counterclockwise[image_] :=ImageRotate[image,Top->Left]

ClearAll[horizontalShift]
horizontalShift[image_,range_] :=counterclockwise[ verticalShift[clockwise[image],range]]

ClearAll[fullShift]
fullShift[image_,range_] := verticalShift[horizontalShift[image,range],range]

ClearAll[shiftRule]
shiftRule[rule_,range_]:= fullShift[Keys[rule],range]-> Values[rule]
SetAttributes[shiftRule,Listable]

shiftRule[temp,30]

Data Augmentation by color jittering

ClearAll[jitterColor]
jitterColor[image_] := Module[{hsbsep=ColorSeparate[ColorConvert[image,"HSB"]]},
hsbsep[[1]]=ImageMultiply[hsbsep[[1]],RandomReal[{0.7,1.4}]];
hsbsep[[1]]=ImageAdd[hsbsep[[1]],RandomReal[{-0.1,0.1}]];
hsbsep[[2]]=ImageMultiply[hsbsep[[1]],RandomReal[{0.7,1.4}]];
hsbsep[[2]]=ImageAdd[hsbsep[[1]],RandomReal[{-0.1,0.1}]];
ColorConvert[ColorCombine[hsbsep,"HSB"],"RGB"]
]
SetAttributes[jitterColor,Listable]

ClearAll[jitterColorRule]
jitterColorRule[rule_] := jitterColor[Keys[rule]]-> Values[rule]
SetAttributes[jitterColorRule,Listable]

jitterColorRule[temp]

Network 1   starts with a large  convolultion network trained to classify natural images, and modifies the final layer from a discrete classification to a measurment of two numbers (Cos and Sin of the rotation angle)

NetReplacePart[Drop[NetModel["SqueezeNet V1.1 Trained on ImageNet Competition Data"],-2],"Input"->NetEncoder[{"Image",{64,64},"ColorSpace"->"RGB"}]]

Normal@Drop[NetModel["SqueezeNet V1.1 Trained on ImageNet Competition Data"],-2]

net=NetChain[Append[
Normal[
NetReplacePart[Drop[NetModel["SqueezeNet V1.1 Trained on ImageNet Competition Data"],-2],"Input"->{3,64,64}]
],
"classifier"->LinearLayer[2]
],
"Input"->NetEncoder[{"Image",{64,64},"ColorSpace"->"RGB"}]
]

The LearningRateMultipliers are set as below to train the last three layers while freezing all of the other layers (nonexecutable code given here since the data is not available)

trainedNet = NetTrain[
	net, 
	trainingSet, 
	ValidationSet -> validationSet, 
	LearningRateMultipliers -> {
		"classifier" -> 1, 
		"conv10" -> 1, 
		"fire9" -> 1, 
		_ -> None}
]

Network 1a   is fine-tuning of network 1.

The LearningRateMultipliers are set to unfreeze the initial layers, but to train them with lower weight:

fineTunedNet = NetTrain[
	trainedNet, 
	trainingSet, 
	ValidationSet -> validationSet, 
	LearningRateMultipliers -> {
		"classifier" -> 1, 
		"conv10" -> 1, 
		"fire9" -> 1, 
		_ -> 0.3
	}
]

Network 2  implements a constraint before the last layer, to dynamically enforce the constraint x^2 +y^1 = 1 on the two components {x,y} = {Cos,Sin}.

The training/validation/test sets are modified by changing {x,y} to {x,y,0}, as illustrated by

rsmod={->{0.9047630485611374`,0.42591528025929865`,0},->{0.9979251503659802`,-0.06438473628924689`,0}};

ClearAll[net,ynet]

ynet=NetGraph[Association[
"part1"->PartLayer[1],
"part2"->PartLayer[2],
"constraint"->ThreadingLayer[#1^2+#2^2-1&],
"reshape"->ReshapeLayer[{1}],
"catenate"->CatenateLayer[]
],
{
"part1"->"constraint",
"part2"->"constraint",
NetPort["Input"]->"catenate",
"constraint"->"reshape"->"catenate"
},
"Input"->2 (* this means the input is 2-component vector *)
]

net=NetChain[Join[
Normal[
NetReplacePart[Drop[NetModel["SqueezeNet V1.1 Trained on ImageNet Competition Data"],-2],"Input"->{3,64,64}]
],
Association[
"classifier"->LinearLayer[2],
"output"->ynet
]
],
"Input"->NetEncoder[{"Image",{64,64},"ColorSpace"->"RGB"}]
]

The network is trained with large learning multipliers on the last 3 layers, and small ones on the initial layers (almost freezing these already trained layers)

trainedNet = NetTrain[
	net ,trainingSet,
	ValidationSet->validationSet,
	LearningRateMultipliers -> {
			"classifier" -> 1,
			"conv10"     -> 0.5,
			"fire9"      -> 0.5,
			"fire8"      -> 0.5,
			"fire7"      -> 0.5,
			_            -> 0.1
		},
	MaxTrainingRounds -> 9
]

Network 2a  sharpens the constraint, by placing a larger multiplier of 4 instead of 1 on the constraint function

ynetSharp=NetGraph[Association[
"part1"->PartLayer[1],
"part2"->PartLayer[2],
"constraint"->ThreadingLayer[4Abs[(#1^2+#2^2-1)]&], (* Multiply constrait by 4 of absolute value  *)
"reshape"->ReshapeLayer[{1}],
"catenate"->CatenateLayer[]
],
{
"part1"->"constraint",
"part2"->"constraint",
NetPort["Input"]->"catenate",
"constraint"->"reshape"->"catenate"
},
"Input"->2 (* this means the input is 2-component vector *)
]

This is appended to the previously trained network 2: 

sharpNet =Import[“constrainedMacNet.wlnet”]

NetChain[Append[Normal[sharpNet],”output”-> ynetSharp],
“Input”->NetEncoder[{“Image”,{64,64},”ColorSpace”->”RGB”}]]

The learning multipliers are set to train the last three layers, and freeze the remaining layers. The number of training rounds is small, but results in a significant change.

tunedSharpNet = NetTrain[
	sharpNet, 
	trainingSet, 
	ValidationSet -> validationSet, 
	LearningRateMultipliers -> {
		"classifier" -> 1, 
		"conv10" -> 1, 
		"fire9" -> 1, 
		"fire8" -> None, 
		"fire7" -> None, 
		_ -> None
	}, 
MaxTrainingRounds -> 3
]

Network 2b  fine-tunes the trained network 2a.

The learning multipliers are set to train all layers, but with larger weights on the last layers, small rates on the initial layers, and intermediate weights on the intervening layers.

tunedSharpNet2alt = NetTrain[
	tunedSharpNet, 
	trainingSet, 
	ValidationSet -> validationSet, 
	LearningRateMultipliers -> {
		"classifier" -> 0.5, 
		"conv10" -> 0.5, 
		"fire9" -> 0.5, 
		"fire8" -> 0.2, 
		"fire7" -> 0.2, 
		_ -> None}, 
	MaxTrainingRounds -> 10
]

It is known that previously trained large convolutional networks can be used as the starting point for new image identifications tasks, using transfer learning and fine-tuning.   In this project we modified a previously trained image classification network to a regression network, retaining the trained initial stages.    The goal was to measure rotations of a 3D object from images, focusing on the special case of rotations about the vertical axis in the image plane.  
 This had some success, but there were complications in that the regression output satisfied a nonlinear constraint which was not enforced by the network, and resulted in more and more drift of the results during subsequent transfer learning and fine-tuning stages.  

The main technique introduced here is the addition of a network layer to dynamically enforce the nonlinear constraint on the output.   Using several stages of transfer learning and fine tuning, this resulted in good results.   We focused on one nonlinear constraint on two-dimensional output data.   The same techniques should generalize to more nonlinear constraints on higher-dimensional output data, for example for the matrices describing the entire group of rotations in 3D space.

The generation of the main visualization was made as follows.   The data corresponding to the correct rotation of a test image is in the list “vin”, the rotation of the same image as measured by the network (network 2b in this case) is in the file “vout”.

vin

vout

ListPlot[{vout,vin},AspectRatio->1]

doubSig=Table[
vold= vin[[k]];
vnew=vout[[k]]/Norm[vout[[k]]];
{Hue[ArcCos[vold.vnew]],Line[{0.75 vold,1.25vnew}]},{k,1,Length[vout]}];

Graphics[{Red,doubSig}]

Hyperlink["Ironman helmet .stl file from thingiverse.com","https://www.thingiverse.com/thing:360747"]

It should be straightforward (but time consuming) to extend this approach to include all rotations, not only those about the vertical axis.  The parameter space is the full rotation group SO(3), which is a 3-dimensional manifold naturally embedded in 9-dimensional Euclidean space.    This will require training a network to output 9 parameters, the 9 components of the rotation matrix, while enforcing the constraints to constrain the resulting 9 numbers to be components of a rotation matrix.   I had  started the data generation as illustrated below, but had not worked on modifying the network appropriately.  There seem to still be some things to learn from fine-tuning the simpler case of rotations about the vertical axis, it seemed better to focus on that than to start this more ambitious generalization prematurely.

Routines for generating training/validation/data sets for full 3D rotation group.  Data augmentation can be performed similarly.

output random SO3 matrix (not leaving any particular axis invariant)

randomSO3:=Module[{O3mat=RandomVariate[CircularRealMatrixDistribution[3]]},Det[O3mat] O3mat]

Generate lists

ClearAll[rasterSO3RotImage]
rasterSO3RotImage[randomSO3_]:=Module[{tempY,viewVec=randomSO3.{0,-1,0}}, 
(*ClearAll[tempY]; *)
tempY=Rasterize[Show[helmetType,ViewPoint->viewVec,Lighting->{
{"Directional",Hue[RandomReal[1]],{frontVec[viewVec],{0,0,0}}},{"Directional",Hue[RandomReal[1]],{frontVec[viewVec],{0,0,0}}},{"Directional",Hue[RandomReal[1]],{frontVec[viewVec],{0,0,0}}}
},ImageSize->imsize],"Image",ImageSize-> {imsize,imsize}]
]

ClearAll[nineComponent]
nineComponent[randomSO3_] :=Transpose[randomSO3]

tempRot=randomSO3;
rasterSO3RotImage[tempRot]-> nineComponent[tempRot]

Initialization

Hyperlink["Best Practices for Fine-tuning Visual Classifiers to New Domains, Brian Chu⋆, Vashisht Madhavan⋆, Oscar Beijbom, Judy Hoffman, Trevor Darrell","http://adas.cvc.uab.es/task-cv2016/papers/0002.pdf"]

Hyperlink["Deep Learning Part 2: Transfer Learning and Fine-tuning Deep Convolutional Neural Networks, Anusua Trivedi","http://blog.revolutionanalytics.com/2016/08/deep-learning-part-2.html"]

Hyperlink["Computer Vision Lab Dresden","http://cvlab-dresden.de/research/scene-understanding/pose-estimation/#DSAC"]

< Neural Networks >

< Rotations >

"Add Timestamp"