I see one thing that could cause the problem right off. different encrypters even using the same algorithm can ofet use different Key Encoding schemes. Java may be using base64 straight style, while .net may be using a proprietery custom scheme. This can be tried by using a custom parser from the byte array output, making the standrad more directly under control.
If this is not the issue, it may be a "tweak" with in standard of the algorithm that causes different structure or processing. With as special as RSA and similar algorithms are I hope not, as this means you may not be able to cross a JAVA and .net set up.
Also, if its within your experience, you can simply unify your approach. Either a Server Side Java Applet, or client side asp/vb.net