You don't have to use Microsoft software or languages to write iPAQ applications - and if you want to then you aren't limited to a single language. There isn't a compiler shipped with PocketPC. You develop on a Windows or Linux box and deploy to the iPAQ.
There are assorted libraries on PocketPC which you can call to do windowing etc., in the same way as there are for Windows. Applications using these will look like the standard PocketPC applications. However, you dont have to use these. All you need is either a compiler that can compile to StrongARM code (MS do them, so do other people. The 'C' people I know use Metrowerk's CodeWarrior) or an appropriate runtime for an interpreted language like Java or C#. Personally I use OTI's j9 Java runtime, 'cos I know java. Works fine for me.
Maybe a place to start looking is
here and especially PocketPC 2000/2002 (as applicable) Software Developers Toolkits.