iPod touch repair

I recently came into possession of 2 ipod touch units. One is a 32Gb 2nd gen, the other an 8Gb 3rd gen (which is essentially same as the 2nd gen).

The first unit had a snapped ribbon cable. The previous owner attempted to replace the touch screen and ended up breaking the cable. To make things worse, the screen looks like it is "white on black". The midframe (plastic stuff) was also missing a big chunk of.

Off to ebay to get:

2nd gen touch screen

mid frame

2nd gen LCD

In the meantime I had a look at the 8Gb 3rd gen unit. Looks like this unit has never been opened, and the only thing really wrong with it is the bent pins in the dock connector. While I waited for the parts to arrive, I thought I'd just take the parts from this unit and get the 32Gb working. I transferred the LCD first, then the touch screen + midframe. After using a lens blower to blow away all the dust and brushing a bit with a lens brush, I closed up the unit. This unit is working very well, but it has a very short battery life. I just then added a battery to the list of parts. I didn't get a chance to take pictures of this unit, but since I moved all the broken parts to the other unit, I will take pictures when I do that.

The 2nd unit is a 8Gb 3rd gen unit (actually a 2nd gen unit).

After removing the LCD, this is what's inside

And after removing the metal shield (desoldered the 2 blobs on both sides and pried up the metal clips) we see the dock connector's completely bent pins

At this point, I was thinking of what to do with it. I could desolder the whole connector and replace it with one from one of my old broken ipod minis. But this means using a SMD rework station to heat the whole area potentially blowing off some of the nearby (tiny) components. I first checked if it was possible to straighten the pins. Using a small screwdriver I carefully "combed" the pins. This is the result:

Using an external USB power supply I plugged it in. No smoke. Seems to be a good sign. I went ahead and hooked up the LCD from the other unit:

That is what I meant when I said the colours were reversed. Everything looked like that, including the apple logo on boot up.

Well, having seen that the board is still working and that the dock connector is somewhat usable, I went ahead and tested it on my computer. It was detected by itunes and wanted to erase the contents. That's a good sign, it means that the USB pins also work. After "combing" the pins some more to get it as straight as I could, I soldered the shield back in place.

Then it's time to unpack the new LCD screen:

After installing the new screen:

After that I had to double sided tape the new frame (which arrived in 3 pieces) to the new touch screen and lock it on.

12-06-10 Update: My batteries for the first unit just arrived (in antistatic bag):

Here it is taken out:

After taking out the old battery:

A test fit:

Applied the solder:

Test with "bad" LCD. I have been using the "reversed colour" LCD to test. I try to keep the working LCD in a dust free environment.

After installing the original LCD and touch screen:

Apple IIe TTL RGB to Commodore 1084s monitor

I have finally finished constructing my cable. Parts have just been sitting there waiting to be put together for a long time but I was too lazy to put them together.

So here is both sides of the cable:

This is how the whole setup looks like (Still no disk drives for now)

When I got this IIe, the Delete key was missing, So I looked for a temporary key to put in its place:

Here's the Startup screen:

Here's some 80 column text

And finally some low res graphics:

Update. After I tried to run some game, I realized that some of the colours look "not-quite-right". Output of the same program on Sweet 16 shows this:

I found from this post that the card actually outputs xrgb. I would have to think of how to resolve this issue... more updates later.

Linux Crash debug tips - I have a soft lockup, what is causing it ?

A soft lockup is the symptom of a task or kernel thread using and not releasing a CPU for a period of time (the softlockup_thresh setting ).  User space processes should not be able to soft lockup a CPU.

How is a soft lockup detected ?

The Linux kernel creates a watchdog process for each CPU in the system.  This should be visible using in the standard "ps" command, this is shown as  [watchdog/N], where N is the number of the logical CPU.

This watchdog process/thread wakes up once a second, gets the current time stamp for the CPU it is running on and saves it into a per-CPU structure. 

The softlockup_tick() function that gets called from the timer interrupt().  This function gets the current time stamp for the specified CPU and compares it to the saved one in the per-CPU structure.

If the current time stamp is more than softlockup_thresh seconds later than the saved time stamp then it's because the watchdog thread has not  been run recently and a soft lockup message is generated on the console. As the timer counter on each CPU can be slightly slow or fast the counter is compared to the previous tick.

Why was the process/task allowed to hog the CPU ?

Not all code that runs on the CPU is considered to be a process.  Tasklets / Interrupt handlers and blah are kernel functions that do not show up in the standard process listing (with the ps command). 

The kernel still allocates them CPU time, but trusts them to relinquish control of the CPU.   

Common causes of not relinquishing control.

Software bugs can cause the process/code to not relinquish the control of the CPU.  The code could be waiting on a lock, or may be running code which continues in an infinite loop.

The other problem may be that the scheduler has ignored the process and the process has not been removed from the CPU for some timeframe.  There are a number of kernel options that can isolate a cpu and run a single process on it.  As the process "hogs" the CPU it may be seen as causing a soft lockup.

Example of a user space "soft lockup"

 Pid: 2642, comm: pausetest Not tainted 2.6.18-164.el5 #1
 RIP: 0033:[<00000000004005c2>]  [<00000000004005c2>]
 RSP: 002b:00007fff8a5d7028  EFLAGS: 00000246
 RAX: 0000000093338370 RBX: 0000000000000028 RCX: 0000000093338340
 RDX: 0000000000000140 RSI: 00002b56f6317000 RDI: 0000000000000001
 RBP: 00002b56f6317000 R08: 00000000ffffffff R09: 000000302f919840
 R10: 0000000000000000 R11: 0000000000000246 R12: 000000302fb51780
 R13: 0000000000000034 R14: 00000000004007da R15: 0000000000000001
 FS:  00002b56f6329210(0000) GS:ffff81001fc147c0(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
 CR2: 000000302f8c5f70 CR3: 000000000db79000 CR4: 00000000000006e0


 Call Trace:

Note the lack of values beneath "Call Trace:" above.  If a userspace process is the process that has not been scheduled/removed from the cpu no backtrace will be shown, only the registers and the process name.


Example of a kernel space "lockup".

BUG: soft lockup detected on CPU#0!

Call Trace:
[] show_trace+0x34/0x47
[] dump_stack+0x12/0x17
[] softlockup_tick+0xdb/0xf6
[] update_process_times+0x42/0x68
[] smp_local_timer_interrupt+0x23/0x47
[] smp_apic_timer_interrupt+0x41/0x47
[] apic_timer_interrupt+0x66/0x6c
DWARF2 unwinder stuck at apic_timer_interrupt+0x66/0x6c
Leftover inexact backtrace:
[] :ext3:htree_dirblock_to_tree+0xa8/0xc9
[] :ext3:htree_dirblock_to_tree+0x72/0xc9
[] :ext3:ext3_htree_fill_tree+0x7c/0x1cb
[] :ext3:ext3_readdir+0x1a9/0x4da
[] filldir+0x0/0xb7
[] file_has_perm+0x94/0xa3
[] filldir+0x0/0xb7
[] vfs_readdir+0x77/0xa9
[] sys_getdents+0x75/0xbd
[] tracesys+0x71/0xdc
[] tracesys+0xd1/0xdc

If a kernel thread is responsible the backtrace for the kernel thread will be shown in the soft lockup message, however the backtrace includes code that shows the stack message.  The DWARF2 undwinder attempts to show the  actual stack trace. (shown above in bold).

Sometimes the call trace may only show the symptom of the soft lockup, and other kernel tasks will need to be investigated to show the cause.

Linux Crash debug tips - I have a partial or "incomplete" vmcore, what can I find out ?

In some circumstances when attempting to debug a vmcore from Linux, you may have only been able to get part of the vmcore, either due a technical issue or the machine being forcebly rebooted before completion.

When loading the vmcore in crash you may find something similar to the message below:

WARNING: yourvmcorefilename: may be truncated or incomplete
         PT_LOAD p_offset: 3358304032
                 p_filesz: 13690204160
           bytes required: 17048508192
            dumpfile size: 5412478976


crash: read error: kernel virtual address: ffff81042ff0eac0  type: "cpu_pda entry"
crash: read error: kernel virtual address: ffff810230005458  type: "pglist node_id"

A workaround to get some data in this case is to use a recent crash and use the --minimal parameter.

crash --minimal vmcorefilehere vmlinux

 --snip --

NOTE: minimal mode commands: log, dis, rd, sym, eval and exit

This will present you with a limited set of commands that you can use that will allow you to get basic data from the vmcore.


It is important to note that backtraces are not entirely accurate and that when you inspect memory it may not be available.  Crash  usually throws an error when this is the case.

This is not an april fools joke.

Linux Crash debug tips - Which process is on which CPU ?

I'm going to start miniblogging (which is larger than twitters microblogging) but not quite enough to be a blog.  For tips that I learn using the crash program.

Say you've had a soft lockup, the kernel has paniced either by your setting or a forced coredump, how can you tell which process is on which CPU.

# crash ./vmcore

>ps 

   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>     0      0   0  ffffffff802e5ae0  RU   0.0       0      0  [swapper]
>     0      1   1  ffff81083006e100  RU   0.0       0      0  [swapper]
>     0      1   2  ffff81010eb26080  RU   0.0       0      0  [swapper]
>     0      1   3  ffff81083e11a080  RU   0.0       0      0  [swapper]
      0      1   4  ffff8104300787a0  RU   0.0       0      0  [swapper]
>     0      1   5  ffff810c3e1d87e0  RU   0.0       0      0  [swapper]
>     0      1   6  ffff81043e15a820  RU   0.0       0      0  [swapper]
>     0      1   7  ffff810c3e200860  RU   0.0       0      0  [swapper]
>     0      1   8  ffff81043e209860  RU   0.0       0      0  [swapper]
>     0      1   9  ffff810c3e2727e0  RU   0.0       0      0  [swapper]
>     0      1  10  ffff81043e21c7e0  RU   0.0       0      0  [swapper]
>     0      1  11  ffff810c3e275860  RU   0.0       0      0  [swapper]
>     0      1  12  ffff81043e2a8820  RU   0.0       0      0  [swapper]
>     0      1  13  ffff810c3e2e87e0  RU   0.0       0      0  [swapper]
>     0      1  14  ffff81043e2b17a0  RU   0.0       0      0  [swapper]
>     0      1  15  ffff810c3e2eb860  RU   0.0       0      0  [swapper]
      1      0  11  ffff81083006b040  UN   0.0   10344    684  init
The tasks that have a > to the left are the processes that are in the RUN state (that means on the CPU right now).

The CPU column aligned to the task is the surprisingly the CPU that the process was on (or was last on).   Too easy.

Replacing a Powermac G4 (Mirror Drive Doors) Power Supply with ATX parts

This is essentially the same thing I previously did with a Powermac G5. A friend of mine acquired a non working Powermac G4 (Mirror Drive Doors) with dual 1.25GHz CPUs. When the power button is pressed, it lights up for a few seconds and goes out. When he saw my post regarding replacing the Powermac G5 power supply, he asked if the same could be done to this unit. The service center was asking for a huge sum to get the power supply replaced and the former owner did not think it was worth it to still get it fixed.

The difference between this and the G5's is that the Power supply case is much smaller. But otherwise could still hold an ATX power supply's innards.

Because my friend has waited over a year for me to arrive (from overseas), we decided to use a good quality name-brand power supply that is rated more than the original. We settled with a Thermaltake Litepower 600w ATX power supply.

After opening up the power supply. I realized that the board is not going to fit easily without any mods, because the heatsinks are made taller than the Apple power supply case. Most probably this is done for better airflow. So in the meantime I proceeded to take the Power Supply apart and desoldered all the wires from the original power supply.

After that, I found a pinout on this site that showed exactly what I needed. I cut the wires on the Thermaltake and soldered the original wires to the remaining wire. This allowed some room to move the wires around so that the original wires can exit through the original hole.

At this point we wanted to test the Mac just to be sure it is still working and that all this trouble isn't for naught.

After plugging it in and using my ADC to DVI cable, we were greeted by the startup "jeng" sound and this appeared on the attached monitor:

Well, we had no hard disks but this is much farther than it ever got on the old power supply. The next step is to mod the heatsinks so that it will all fit into the original enclosure. After that we tie the cables neatly into bundles with zip tie and put it back into the original case. We cut some loose cintra boards and used double sided tape to attach them under the board to protect it from shorts.

Then finally mount it into the tower case.

Plugging in the hard disk from an old B&W G3 finally boots up

Most of the stuff works. The 2 things that doesn't are: The ADC port cannot power an external ADC display anymore. The Firewire ports don't have power (Not sure if a self-powered firewire device would work). This is because ATX does not have the 25v needed for those 2 to function. If we add a 25v power supply (into the empty space in the power supply) we should be able to get these working again.

Repairing a membrane type apple keyboard

They don't make keyboards anymore like they used to. In the past each key of a keyboard had a keyswitch under it. Every keyboard Apple made from the original Apple ][ ... up to the ADB Extended Keyboard II were made in this manner. (there may be others, but I have not come across them).

The first Apple USB keyboard I have is the one that came bundled with the imac G3. It was bondi blue, had a USB hub with 2 ports on both sides of the keyboard. It also (like the ADB keyboards that came before it) had a power button. I don't quite remember how this keyboard came into my possession, but when I tried to use it, some buttons wouldn't work. Particularly, the ESC key and probably the first few function keys next to it.

Getting this unit open wasn't too hard: just have some patience and remove 50 something phillips head screws. What got my attention was that there are no longer keyswitches under each key. (not in the sense as in my description above anyway). It has been replaced with 3 sheets of what looks like "acetate transparency sheets" we used to use to project presentations. the top and bottom sheets had some circuitry on it, very similar to a flexible PCB. The middle sheet had holes punched on it. By now you'd have figured out the rest. The user would "press" a key and those keys would line up where the holes are. Top sheet contacts with bottom sheet and voila the key is "pressed"

Unlike PCBs, the "circuitry" on the top and bottom sheets seem very fragile. The former owner may have spilled some cola into it (or maybe just water). Parts of this "circuitry" is now a different colour from the rest. Poking my ohmmeter between 2 points that's supposed to conduct shows no connection.

My original solution to this was to take some aluminum foil. carefully cut it to the right size, and then along with some cardboard, get some nearby screws to "press down" on this. This solution did work.... for a while... eventually the cardboard would flatten out and what was a tight fit is now loose... sometimes the keys wouldn't connect anymore. I have opened this unit up again to replace the cardboard.

Now... many months after that, a good friend of mine acquired a powermac G5 and suspected that the power supply may not be good. While working on this unit, he let me use a white Apple USB keyboard (not the aluminum ones)... I was trying to get front row to appear by pressing the keyboard shortcut command-ESC. This did not work. After some more testing, I worked out that the ESC key, along with F1 - F6 are not working. I was ready to try my aluminum foil solution again. Unfortunately for me, the connection was broken over a wide area .. almost 2 inches wide. I could not get the foil and cardboard to be pressed down enough to make a lasting connection. Then I remembered:

Surely I can use this to repair the dissolved circuitry. A trip to the local electronics store snagged me the last unit in stock.

I had another identical keyboard at home waiting to be fixed. This is how the membranes looked like after opening. Note that the fluid has seeped up deep into the keyboard due to capillary action:

After using some baby wipes to clean out the fluid, you can see the damage that was done:

In the above picture, some of the lines are darker. Testing these with my ohmmeter reveal that they no longer conduct. After using the conductive pen on it and being careful not to bridge connections that aren't supposed to be:

An xacto knife can be handy in scraping off accidentally bridged connections. After letting the ink dry (did not take more than 5 minutes), I tested the unit before putting all the screws back. I also took the opportunity to clean the insides of some nasty looking stuff (mostly crumbs and spilled drinks). Keyboard is all working again.