Since version 2.90d, almost all GregTech machines can accept upgrades.
If a machine can be upgraded, its tooltip will contain the label "Possible Upgrades:" followed by a series of letters, reflecting the upgrades the machine will accept. Not every machine can accept every upgrade.

Label

Upgrade Accepted
O
 Overclocker
T
 Transformer
B
 Battery (Energy Storage)
M
 MJ (Pneumatic Generator)
S
 Steam (Steam Engine)
Upgrades are installed by holding the Upgrade item in your hand, and right clicking the machine you want to upgrade.
Machines that accept upgrades can only accept 16 upgrades total.
Overclocker and Transformer upgrade items in inventory stack up to 4. (configurable)
Some upgrades are crafted at a crafting table, while some must be crafted via the Assembling Machine (Assembler).
Upgrades cannot be removed once installed. [This can be bypassed for automatic machines, by enabling the conversion to IC² machines in the config.]
Machines, when wrenched and retrieved as items, will retain their upgrades, and their tooltips will state the installed upgrades.
The GregTech Portable Scanner can also show what upgrades a machine has installed.

Energy Storage upgrades :

Type
 Internal storage bonus
 Recipe
 Assembler
Energy Storage*
 10k EU
 1x RE-Battery, 1x Electronic Circuit, 2x Ins. Copper Cable, 5x Wooden Planks
 No
Energy Crystal
 100k EU
 1x Energy Crystal, 1x Refined Iron/Aluminium Plate
 Yes
Lithium Battery
 100k EU
 1x Lithium Battery, 1x Refined Iron/Aluminium Plate
 Yes
Lapotron Crystal
 1M EU
 1x Lapotron Crystal, 1x Refined Iron/Aluminium Plate
 Yes
Energy Orb
 10M EU
 1x Energy Orb, 1x Refined Iron/Aluminium Plate
 Yes
Redstone Energy Cell
 100k MJ
 1x Energy Cell Frame (Full) [TE], 1x Energy Storage Upgrade -> 4x upgrades
 Yes
Steam Tank
 64 Buckets
 2x Steel plate, 1x Glass pane
 Yes
*vanilla IC² upgrade
Note: As of version 4.03d, all EU storage upgrades above most basic 10k, have 10x their original values. So, Energy Crystal and Lithium Battery store 1M, Lapotron Crystal stores 10M, and Energy Orb stores 100M EU. This is based off of a change made in IC2 Experimental that created an additional tier between the Batbox and MFE.
There is no limit on how many of each individual energy storage upgrade can be installed to a machine (except for the 16 upgrade total cap).
Energy stored in the different storage types (EU, MJ, Steam) cannot be inter-converted, and machines will only output EU. However, if a machine contains MJ or Steam, it will consume them before it starts consuming EU.
Note: Higher tier EU storage upgrades can only be applied if the machine has enough Transformer upgrades.
For example, in order to accept an Energy Crystal upgrade, the machine must be able to accept HV.
In order to accept a Lapotron Crystal upgrade, the machine must be able to accept EV.

Other Upgrades :

Type
 Effect
 Recipe
 Assembler
Overclocker*
 Operation time x1/2, total EU usage x2, EU/t required to operate x4
 3x Coolant Cell, 1x Electronic Circuit, 2x Ins. Copper Cable
 No

Overclocker Alternate recipe ------------------------------------------------>
 Replace 3x Coolant Cell with 1x NaK/Helium Coolant cell -> 2x Overclockers
 No
Transformer*
 Maximum EU/p input x4, EU/t output x4, 1st upgrade outputs 4 packets
 1x MV Transformer, 1x Electronic Circuit, 2x 2xIns. Gold Cable
 No
HV-Transformer
 Same as above, but required for the last two transformer upgrades.
 1x HV Transformer, 1x Transformer Upgrade
 Yes
Pneumatic Generator
 Allows the machine to use MJ as an energy source (1 MJ -> 1 EU)
NO LONGER WORKS - MJ SUPPORT DROPPED
 1x Generator, 1x Iron/Steel Gear (BC/RC)
 Yes
Steam
 Allows the machine to use Steam as an energy source (2 mB -> 1 EU)
 1x Industrial Steam Engine (RC), 2x Steel Plate
 Yes
*vanilla IC² upgrade
Unlike the energy storage upgrades, each machine is limited to having:
Note: If a machine is overclocked too much, it may require more EU/t than its default storage capacity. To compensate for this, add an Energy Storage upgrade.
Note 2: 4 Overclocker Upgrades will increase a machine's required EU/t to 256x its original value. So, a 4x-overclocked Blast Furnace would consume 32,768EU/t.


Utility Upgrades :
Type
 Effect
 Recipe
Kanthal Heating Coil
 Increases efficiency of Electric Furnaces (GT only) by 100%
 4 Kanthal Ingots in a Wiremill
Nichrome Heating Coil
 Increases efficiency of Electric Furnaces (GT only) by 50%
 5 Nichrome Ingots in a Wiremill
Kanthal Heating Coil x4
 Increases Industrial Blast Furnace Heat by 500K
 4 Kanthal Ingots in a Wiremill
Nichrome Heating Coil x4
 Increases Industrial Blast Furnace Heat by 500K
 5 Nichrome Ingots in a Wiremill
Note: Kanthal Heating Coil(s) must be applied before Nichrome Heating Coil(s).
An Automatic E-Furnace with a Kanthal Heating Coil installed will perform operations in 50% of their normal time for 50% of their normal energy (200% efficiency).
An Automatic E-Furnace with both a Kanthal and a Nichrome Heating Coil installed will perform operations in 33% of their normal time for 33% of their normal energy (300% efficiency).
Alloy Smelters can be upgraded in the same way Automatic E-Furnaces can.

Utility Upgrades can each be used once per machine.

As of v4.04a, Kanthal and Nichrome ingots were added, and the Heating Coil recipes were transferred to the Wiremill. Previously, Kanthal Heating Coils were crafted with 3 (Refined) Iron Ingots, 3 Chrome Ingots, 3 Aluminium Ingots -> 3x Kanthal Heating Coils, and Nichrome Heating Coils were crafted with 4 Nickel Ingots and 1 Chrome Ingot. If Railcraft was installed, both of these recipes took place in the Rolling Machine.


In Gregtech version 4.06c, more annoying sounds were added to machines. These sounds are rather loud and annoying after a while, but can be turned off on a machine by putting a Muffler Upgrade on the machine. Items with sounds like the Debug and Portable Scanner have sounds which can't be stopped.
Type
 Effect
 Recipe
Muffler Upgrade
 Stops any Gregtech machine making sounds
 1x Iron/Aluminium Plate and 2x Wood Pulp in an Assembler