You might ask, "Aren't there APIs to which I have to pass a pointer?" You are right. There are not only a few such APIs, but quite a large number (a small understatement). This access to native WIN32 code sometimes makes using unsafe classic pointers mandatory (although some of it can be handled by the support of COM and PInvoke).
Although the default for C# code is safe mode, you can declare certain classes or only methods of classes to be unsafe. This declaration enables you to use pointers, structs, and statically allocated arrays. Both safe code and unsafe code run in the managed space, which implies that no marshaling is incurred when calling unsafe code from safe code.
What are the implications of dealing with your own memory in unsafemode? Well, the garbage collector, of course, may not touch your memory locations and move them just as it does for managed code. Unsafe variables are pinned into the memory block managed by the garbage collector.