warning: unable to rmdir my/module: Directory not empty

I am getting a bizarre behavior with git:

#!/bin/sh

# master
cd /tmp
mkdir test
cd test
git init
touch README
git add README
git commit -mREADME

# submodule
cd ..
mkdir module
cd module
git init
touch MODULE
git add MODULE
git commit -mMODULE

cd ../test
git submodule add file:///tmp/module my/module
git commit -a -m"adding submodule"

# empty branch
git symbolic-ref HEAD refs/heads/newbranch
rm .git/index
git clean -fdx
touch EMPTY
git add EMPTY
git commit -a -mEMPTY

# update
git checkout master
git submodule init
git submodule update
git checkout newbranch