frustum correction / "portal view"

[Tech B]

I've always loved games and decided to pick up dev as a new hobby. I'm aiming to do something like Johnny Lee and track head position for a VR display. I've heard it called frustum correction or the "portal effect". Here is a link to see what I'm talking about.

Could some one explain how a 3d program applies mapped head positions to the 3d environment. I'm using Blender4.9 and Python2.6, could this be done in Blender, or will I have to code something in Panda3d?

I wrote some code to take care of the tracking. I can track faces, single and multiple. I can also track up to 2 IR LED's

I understand how the effect works in a general sense; just not the math and technical overview involved.

[Tech B]

I switched to Panda3D so I could code everything I needed. Blenders Game Engine would have been my ideal environment to do this in, but I can model everything and export it to a .egg file for Panda3D to read.

For more details on how I set up the webcam ckeck this out.

I will try to post a video sometime of what I've done; for now here is the code.

This is the Panda3D code

#Panda3d Head (IR)tracking
#K.B. Carte - Tech B.
#Working Proof of Concept
#6/17/10

#things needed
import IRClass
import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject
from pandac.PandaModules import *

ir = IRClass.IR()

def irTask(task):

    try:
        #get x,y of IR dot scaled to the window size. Currently 1000,800
        x,y = ir.xySinglePos(scale=(base.win.getProperties().getXSize(),
                                    base.win.getProperties().getYSize()))

        #This is based on 0,0 being the center of the screen
        #So if we wanted to move the perspective left, it needs to be negitave
        x = ((base.win.getProperties().getXSize()/2) - x) * 0.01
        y = ((base.win.getProperties().getYSize()/2) - y) * 0.01

        #print x,y
        #set the camera's posistion in a 2D way, hence z = y
        camera.setPos(x,0,y)
    #x,y return None if no IR was dectected, so we pass
    except TypeError:
        pass
    #needed to repeat every frame
    return task.cont

#disable so we can set the cameras pos
base.disableMouse()
#add light so we see textures
light = DirectionalLight('light')
lightnp = render.attachNewNode(light)
render.setLight(lightnp)

#something to focus on for the perspective effect
plane = loader.loadModel("BG")
plane.reparentTo(render)
plane.setPos(0,100,0)
#an object to view to see if its working
box = loader.loadModel("box")
box.reparentTo(render)
box.setPos(0,30,0)
# always point to camera at the plane
camera.lookAt(plane)
#add the IR tracking task
taskMgr.add(irTask,'ir_task')
#run forever
run()

This is the IRClass for ir tracking:

#IRClass.py
#K.B. Carte - Tech B.
#Class to handle IR tracking.
#Modules used:
#VideoCapture - http://videocapture.sourceforge.net/
#PIL - http://www.pythonware.com/products/pil/
#Note: VideoCapture doesn't work with all webcams
#All models where made in Blender and exported to .egg
#format via Chicken Exporter.

from VideoCapture import Device
import Image
from ctypes import *


#Returns size in pixels of the ir dot
#not needed yet; may be used for distance
def irint(im):
  xyi = 0
  irxy = im.getprojection()
  irx = irxy[0]
  iry = irxy[1]

  for i in irx:
    if i == 1:
      xyi +=1
  for i in iry:
    if i == 1:
      xyi += 1
  return xyi


class IR:
    """Class for tracking up to two IR dots. Still a work in progress.
    This was ment to be used in Panda3D, but could find usefull in other places.
    irDots() isn't that accurate, and was sped up by increasing x += and y += don't
    increase it too much or the results will be less than pleasing."""
    def __init__(self):
        """self.res will soon take an image instance and get the width and
        height from it; so it will be webcam independent. """
        self.cam = Device()
        self.user = windll.user32
        self.res = (640,480)

    class POINT(Structure):
        """Pointer to get current mouse position from Windows system"""
        _fields_ = [("x", c_long),("y", c_long)]

    def irSingle(self,im):
        """Track single IR dot using PIL.Image.getprojection.
        Finds the first true pixel value."""
        imxy = im.getprojection()
        imx = imxy[0]
        imy = imxy[1]
        x = imx.index(1)
        y = imy.index(1)
        return (x,y)

    def irDots(self,im):
        """Track up to 2 dots. This method is sped up by increments of 4.
        This method is also not that accurate and is jumpy/jerky."""
        imxy = []
        x = 0
        y = 0
        while x*y <= self.res[0]*self.res[1]:
            if im.getpixel((x,y)) >= (50,50,50):
                imxy.append((x,y))
            x += 4
            if x == 640:
                x = 0
                y += 4
            if y == 480:
              return imxy

    def xyDotPos(self,scale=(1440,900)):
        """Limited to two dots.
        Scale the image instance and return x,y of the IR dots.
        default scale is the resolution of my moniter"""
        try:
            im = self.cam.getImage()
            im.resize(scale)
            xy = IR.irDots(self,im)
            return (xy[0],xy[-1])
        except:
            pass

    def xySinglePos(self, scale=(1440,900)):
        """Scale the image instance and find x,y of single IR dot."""
        try:
            im = self.cam.getImage()
            im.resize(scale)
            x,y = IR.irSingle(self,im)
            return (x,y)
        except ValueError:
            pass

    def setPos(self,x,y):
        """Set cursor posistion. Windows only.
        This is the virtual cursor you see, not the system cursor. So programs
        that use the system cursor will override this method. Example would be some
        videogames, mainly first person shooters."""
        self.user.SetCursorPos(x,y)