Memory leak when making CGImage from MTLTexture (Swift, macOS)

I have a Metal app and I'm trying to export frames to a quicktime movie. I am rendering frames in super hi-res and then scaling them down before writing, in order to antialias the scene.

To scale it, I'm taking the hi-res texture and converting it to a CGImage, then I resize the image and write out the smaller version. I have this extension I found online for converting an MTLTexture to a CGImage:

extension MTLTexture

func bytes() -> UnsafeMutableRawPointer
let width = self.width
let height = self.height
let rowBytes = self.width * 4
let p = malloc(width * height * 4)

self.getBytes(p!, bytesPerRow: rowBytes, from: MTLRegionMake2D(0, 0, width, height), mipmapLevel: 0)

return p!


func toImage() -> CGImage? CGBitmapInfo.byteOrder32Little.rawValue // noneSkipFirst
let bitmapInfo:CGBitmapInfo = CGBitmapInfo(rawValue: rawBitmapInfo)

let size = self.width * self.height * 4
let rowBytes = self.width * 4

let releaseMaskImagePixelData: CGDataProviderReleaseDataCallback = (info: UnsafeMutableRawPointer?, data: UnsafeRawPointer, size: Int) -> () in
// https://developer.apple.com/reference/coregraphics/cgdataproviderreleasedatacallback
// N.B. 'CGDataProviderRelease' is unavailable: Core Foundation objects are automatically memory managed
return

if let provider = CGDataProvider(dataInfo: nil, data: p, size: size, releaseData: releaseMaskImagePixelData)

let cgImageRef = CGImage(width: self.width, height: self.height, bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: rowBytes, space: pColorSpace, bitmapInfo: bitmapInfo, provider: provider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent)!

return cgImageRef

return nil


// end extension


I'm not positive, but it seems like something in this function is resulting in the memory leak -- with every frame it is holding on to the amount of memory in the giant texture / cgimage and not releasing it.

The CGDataProvider initialization takes that 'releaseData' callback argument, but I was under the impression that it was no longer needed.

I also have a resizing extention to CGImage -- this might also cause a leak, I don't know. However, I can comment out the resizing and writing of the frame, and the memory leak still builds up, so it seems to me that the conversion to CGImage is the main problem.

extension CGImage

func resize(_ scale:Float) -> CGImage?

let imageWidth = Float(width)
let imageHeight = Float(height)

let w = Int(imageWidth * scale)
let h = Int(imageHeight * scale)

guard let colorSpace = colorSpace else return nil
guard let context = CGContext(data: nil, width: w, height: h, bitsPerComponent: bitsPerComponent, bytesPerRow: Int(Float(bytesPerRow)*scale), space: colorSpace, bitmapInfo: alphaInfo.rawValue) else return nil

// draw image to context (resizing it)
context.interpolationQuality = .high
let r = CGRect(x: 0, y: 0, width: w, height: h)
context.clear(r)
context.draw(self, in:r)

// extract resulting image from context
return context.makeImage()





Finally, here is the big function that I call every frame when exporting. I'm sorry for the length but it is probably better to provide too much information than too little. So, basically at the start of rendering I allocate a giant MTL texture ('exportTextureBig'), the size of my normal screen multiplied by 'zoom_subvisions' in each direction. I render the scene in chunks, one for each spot on the grid, and assemble the large frame by using blitCommandEncoder.copy() to copy each small chunk onto the large texture. Once the entire frame is filled in, then I try to make a CGImage from it, scale it down to another CGImage, and write that out.

I'm calling commandBuffer.waitUntilCompleted() every frame while exporting -- hoping to avoid having the renderer hold on to textures that it is still using.

func exportFrame2(_ commandBuffer:MTLCommandBuffer, _ texture:MTLTexture) // texture is the offscreen render target for the screen-size chunks

if zoom_index < zoom_subdivisions*zoom_subdivisions // copy screen-size chunk to large texture

if let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder()

let dx = Int(BigRender.globals_L.displaySize.x) * (zoom_index%zoom_subdivisions)
let dy = Int(BigRender.globals_L.displaySize.y) * (zoom_index/zoom_subdivisions)
blitCommandEncoder.copy(from:texture,
sourceSlice: 0,
sourceLevel: 0,
sourceOrigin: MTLOrigin(x:0,y:0,z:0),
sourceSize: MTLSize(width:Int(BigRender.globals_L.displaySize.x),height:Int(BigRender.globals_L.displaySize.y), depth:1),
to:BigVideoWriter!.exportTextureBig!,
destinationSlice: 0,
destinationLevel: 0,
destinationOrigin: MTLOrigin(x:dx,y:dy,z:0))

blitCommandEncoder.synchronize(resource: BigVideoWriter!.exportTextureBig!)
blitCommandEncoder.endEncoding()





commandBuffer.commit()
commandBuffer.waitUntilCompleted() // do this instead

// is big frame complete?
if (zoom_index == zoom_subdivisions*zoom_subdivisions-1)

// shrink the big texture here

if let cgImage = self.exportTextureBig!.toImage() // memory leak here?

// this can be commented out and memory leak still happens
if let smallImage = cgImage.resize(1.0/Float(zoom_subdivisions))
writeFrame(nil, smallImage)









This all works, except for the huge memory leak. Is there something I can do to make it release the cgImage data each frame? Why is it holding onto it?

Thanks very much for any suggestions!

1 Answer
1

I think you've misunderstood the issue with CGDataProviderReleaseDataCallback and CGDataProviderRelease() being unavailable.

CGDataProviderReleaseDataCallback

CGDataProviderRelease()

CGDataProviderRelease() is (in C) used to release the CGDataProvider object itself. But that's not the same thing as the byte buffer that you've provided to the CGDataProvider when you created it.

CGDataProviderRelease()

CGDataProvider

CGDataProvider

In Swift, the lifetime of the CGDataProvider object is managed for you, but that doesn't help deallocate the byte buffer.

CGDataProvider

Ideally, CGDataProvider would be able to automatically manage the lifetime of the byte buffer, but it can't. CGDataProvider doesn't know how to release that byte buffer because it doesn't know how it was allocated. That's why you have to provide a callback that it can use to release it. You are essentially providing the knowledge of how to release the byte buffer.

CGDataProvider

CGDataProvider

Since you're using malloc() to allocate the byte buffer, your callback needs to free() it.

malloc()

free()

That said, you'd be much better off using CFMutableData rather than UnsafeMutableRawPointer. Then, create the data provider using CGDataProvider(data:). In this case, all of the memory is managed for you.

CFMutableData

UnsafeMutableRawPointer

CGDataProvider(data:)

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.